I am working on the file "0.24/_downloads/e23ed246a9a354f899dfb3ce3b06e194/10_overview.ipynb".
The first task is:
We'll also pass the event dictionary as the event_id parameter (so we can
work with easy-to-pool event labels instead of the integer event IDs), and
specify tmin and tmax (the time relative to each event at which to
start and end each epoch). As mentioned above, by default ~mne.io.Raw and
~mne.Epochs data aren't loaded into memory (they're accessed from disk only
when needed), but here we'll force loading into memory using the
preload=True parameter so that we can see the results of the rejection
criteria being applied:
Can you write Python code for it?

epochs = mne.Epochs(raw, events, event_id=event_dict, tmin=-0.2, tmax=0.5,
                    reject=reject_criteria, preload=True)


I am working on the file "0.24/_downloads/e23ed246a9a354f899dfb3ce3b06e194/10_overview.ipynb".
The first task is:
~mne.io.Raw objects also have several built-in plotting methods; here we
show the power spectral density (PSD) for each sensor type with
~mne.io.Raw.plot_psd, as well as a plot of the raw sensor traces with
~mne.io.Raw.plot. In the PSD plot, we'll only plot frequencies below 50 Hz
(since our data are low-pass filtered at 40 Hz). In interactive Python
sessions, ~mne.io.Raw.plot is interactive and allows scrolling, scaling,
bad channel marking, annotations, projector toggling, etc.
Can you write Python code for it?


I am working on the file "0.24/_downloads/e41b6a898e7a75f8a9f1a6c00ca73857/20_visualize_epochs.ipynb".
The first task is:
If you prefer untransformed power estimates, you can pass dB=False. It is
also possible to normalize the power estimates by dividing by the total power
across all frequencies, by passing normalize=True. See the docstring of
~mne.Epochs.plot_psd_topomap for details.
Plotting Epochs as an image map
A convenient way to visualize many epochs simultaneously is to plot them as
an image map, with each row of pixels in the image representing a single
epoch, the horizontal axis representing time, and each pixel's color
representing the signal value at that time sample for that epoch. Of course,
this requires either a separate image map for each channel, or some way of
combining information across channels. The latter is possible using the
~mne.Epochs.plot_image method; the former can be achieved with the
~mne.Epochs.plot_image method (one channel at a time) or with the
~mne.Epochs.plot_topo_image method (all sensors at once).
By default, the image map generated by ~mne.Epochs.plot_image will be
accompanied by a scalebar indicating the range of the colormap, and a time
series showing the average signal across epochs and a bootstrapped 95%
confidence band around the mean. ~mne.Epochs.plot_image is a highly
customizable method with many parameters, including customization of the
auxiliary colorbar and averaged time series subplots. See the docstrings of
~mne.Epochs.plot_image and mne.viz.plot_compare_evokeds (which is
used to plot the average time series) for full details. Here we'll show the
mean across magnetometers for all epochs with an auditory stimulus:
Can you write Python code for it?


I am working on the file "0.24/_downloads/e41b6a898e7a75f8a9f1a6c00ca73857/20_visualize_epochs.ipynb".
The first task is:
Just as we saw in the tut-section-raw-plot-proj section, we can plot
the projectors present in an ~mne.Epochs object using the same
~mne.Epochs.plot_projs_topomap method. Since the original three
empty-room magnetometer projectors were inherited from the
~mne.io.Raw file, and we added two ECG projectors for each sensor
type, we should see nine projector topomaps:
Can you write Python code for it?


I am working on the file "0.24/_downloads/e41b6a898e7a75f8a9f1a6c00ca73857/20_visualize_epochs.ipynb".
The first task is:
Just like ~mne.Epochs.plot_projs_topomap,
~mne.Epochs.plot_psd_topomap has a vlim='joint' option for fixing
the colorbar limits jointly across all subplots, to give a better sense of
the relative magnitude in each frequency band. You can change which channel
type is used  via the ch_type parameter, and if you want to view
different frequency bands than the defaults, the bands parameter takes a
list of tuples, with each tuple containing either a single frequency and a
subplot title, or lower/upper frequency limits and a subplot title:
Can you write Python code for it?

bands = [(10, '10 Hz'), (15, '15 Hz'), (20, '20 Hz'), (10, 20, '10-20 Hz')]
epochs['visual/right'].plot_psd_topomap(bands=bands, vlim='joint',
                                        ch_type='grad')


I am working on the file "0.24/_downloads/e41b6a898e7a75f8a9f1a6c00ca73857/20_visualize_epochs.ipynb".
The first task is:
Note that these field maps illustrate aspects of the signal that have
already been removed (because projectors in ~mne.io.Raw data are
applied by default when epoching, and because we called
~mne.Epochs.apply_proj after adding additional ECG projectors from
file). You can check this by examining the 'active' field of the
projectors:
Can you write Python code for it?

print(all(proj['active'] for proj in epochs.info['projs']))


I am working on the file "0.24/_downloads/e41b6a898e7a75f8a9f1a6c00ca73857/20_visualize_epochs.ipynb".
The first task is:
Plotting Epochs as time series
.. sidebar:: Interactivity in pipelines and scripts
To use the interactive features of the `~mne.Epochs.plot` method
when running your code non-interactively, pass the ``block=True``
parameter, which halts the Python interpreter until the figure window is
closed. That way, any channels or epochs that you mark as "bad" will be
taken into account in subsequent processing steps.

To visualize epoched data as time series (one time series per channel), the
mne.Epochs.plot method is available. It creates an interactive window
where you can scroll through epochs and channels, enable/disable any
unapplied :term:SSP projectors <projector> to see how they affect the
signal, and even manually mark bad channels (by clicking the channel name) or
bad epochs (by clicking the data) for later dropping. Channels marked "bad"
will be shown in light grey color and will be added to
epochs.info['bads']; epochs marked as bad will be indicated as 'USER'
in epochs.drop_log.
Here we'll plot only the "catch" trials from the sample dataset
<sample-dataset>, and pass in our events array so that the button press
responses also get marked (we'll plot them in red, and plot the "face" events
defining time zero for each epoch in blue). We also need to pass in
our event_dict so that the ~mne.Epochs.plot method will know what
we mean by "button" — this is because subsetting the conditions by
calling epochs['face'] automatically purges the dropped entries from
epochs.event_id:
Can you write Python code for it?

catch_trials_and_buttonpresses = mne.pick_events(events, include=[5, 32])
epochs['face'].plot(events=catch_trials_and_buttonpresses, event_id=event_dict,
                    event_color=dict(button='red', face='blue'))


I am working on the file "0.24/_downloads/e41b6a898e7a75f8a9f1a6c00ca73857/20_visualize_epochs.ipynb".
The first task is:
Plotting sensor locations
Just like ~mne.io.Raw objects, ~mne.Epochs objects
keep track of sensor locations, which can be visualized with the
~mne.Epochs.plot_sensors method:
Can you write Python code for it?

epochs.plot_sensors(kind='3d', ch_type='all')
epochs.plot_sensors(kind='topomap', ch_type='all')


I am working on the file "0.24/_downloads/e41b6a898e7a75f8a9f1a6c00ca73857/20_visualize_epochs.ipynb".
The first task is:
To plot an image map for all sensors, use
~mne.Epochs.plot_topo_image, which is optimized for plotting a large
number of image maps simultaneously, and (in interactive sessions) allows you
to click on each small image map to pop open a separate figure with the
full-sized image plot (as if you had called ~mne.Epochs.plot_image on
just that sensor). At the small scale shown in this tutorial it's hard to see
much useful detail in these plots; it's often best when plotting
interactively to maximize the topo image plots to fullscreen. The default is
a figure with black background, so here we specify a white background and
black foreground text. By default ~mne.Epochs.plot_topo_image will
show magnetometers and gradiometers on the same plot (and hence not show a
colorbar, since the sensors are on different scales) so we'll also pass a
~mne.channels.Layout restricting each plot to one channel type.
First, however, we'll also drop any epochs that have unusually high signal
levels, because they can cause the colormap limits to be too extreme and
therefore mask smaller signal fluctuations of interest.
Can you write Python code for it?

reject_criteria = dict(mag=3000e-15,     # 3000 fT
                       grad=3000e-13,    # 3000 fT/cm
                       eeg=150e-6)       # 150 µV
epochs.drop_bad(reject=reject_criteria)

for ch_type, title in dict(mag='Magnetometers', grad='Gradiometers').items():
    layout = mne.channels.find_layout(epochs.info, ch_type=ch_type)
    epochs['auditory/left'].plot_topo_image(layout=layout, fig_facecolor='w',
                                            font_color='k', title=title)


I am working on the file "0.24/_downloads/e41b6a898e7a75f8a9f1a6c00ca73857/20_visualize_epochs.ipynb".
The first task is:
To plot image maps for all EEG sensors, pass an EEG layout as the layout
parameter of ~mne.Epochs.plot_topo_image. Note also here the use of
the sigma parameter, which smooths each image map along the vertical
dimension (across epochs) which can make it easier to see patterns across the
small image maps (by smearing noisy epochs onto their neighbors, while
reinforcing parts of the image where adjacent epochs are similar). However,
sigma can also disguise epochs that have persistent extreme values and
maybe should have been excluded, so it should be used with caution.
Can you write Python code for it?

layout = mne.channels.find_layout(epochs.info, ch_type='eeg')
epochs['auditory/left'].plot_topo_image(layout=layout, fig_facecolor='w',
                                        font_color='k', sigma=1)


I am working on the file "0.24/_downloads/e41b6a898e7a75f8a9f1a6c00ca73857/20_visualize_epochs.ipynb".
The first task is:
To plot image maps for individual sensors or a small group of sensors, use
the picks parameter. Passing combine=None (the default) will yield
separate plots for each sensor in picks; passing combine='gfp' will
plot the global field power (useful for combining sensors that respond with
opposite polarity).
Can you write Python code for it?

epochs['auditory'].plot_image(picks=['MEG 0242', 'MEG 0243'])
epochs['auditory'].plot_image(picks=['MEG 0242', 'MEG 0243'], combine='gfp')


I am working on the file "0.24/_downloads/e41b6a898e7a75f8a9f1a6c00ca73857/20_visualize_epochs.ipynb".
The first task is:
Visualizing epoched data
This tutorial shows how to plot epoched data as time series, how to plot the
spectral density of epoched data, how to plot epochs as an imagemap, and how to
plot the sensor locations and projectors stored in ~mne.Epochs
objects.
We'll start by importing the modules we need, loading the continuous (raw)
sample data, and cropping it to save memory:
Can you write Python code for it?

import os
import mne

sample_data_folder = mne.datasets.sample.data_path()
sample_data_raw_file = os.path.join(sample_data_folder, 'MEG', 'sample',
                                    'sample_audvis_raw.fif')
raw = mne.io.read_raw_fif(sample_data_raw_file, verbose=False).crop(tmax=120)


I am working on the file "0.24/_downloads/e7de7baffeeb4beff147bad8657b46dc/opm_data.ipynb".
The first task is:
Examine our coordinate alignment for source localization and compute a
forward operator:

Note

The Head<->MRI transform is an identity matrix, as the
          co-registration method used equates the two coordinate
          systems. This mis-defines the head coordinate system
          (which should be based on the LPA, Nasion, and RPA)
          but should be fine for these analyses.

Can you write Python code for it?

bem = mne.read_bem_solution(bem_fname)
trans = mne.transforms.Transform('head', 'mri')  # identity transformation

# To compute the forward solution, we must
# provide our temporary/custom coil definitions, which can be done as::
#
# with mne.use_coil_def(coil_def_fname):
#     fwd = mne.make_forward_solution(
#         raw.info, trans, src, bem, eeg=False, mindist=5.0,
#         n_jobs=1, verbose=True)

fwd = mne.read_forward_solution(fwd_fname)
# use fixed orientation here just to save memory later
mne.convert_forward_solution(fwd, force_fixed=True, copy=False)

with mne.use_coil_def(coil_def_fname):
    fig = mne.viz.plot_alignment(evoked.info, trans=trans, subject=subject,
                                 subjects_dir=subjects_dir,
                                 surfaces=('head', 'pial'), bem=bem)

mne.viz.set_3d_view(figure=fig, azimuth=45, elevation=60, distance=0.4,
                    focalpoint=(0.02, 0, 0.04))


I am working on the file "0.24/_downloads/e7de7baffeeb4beff147bad8657b46dc/opm_data.ipynb".
The first task is:
Perform minimum-norm localization
Due to the small number of sensors, there will be some leakage of activity
to areas with low/no sensitivity. Constraining the source space to
areas we are sensitive to might be a good idea.
Can you write Python code for it?

inverse_operator = mne.minimum_norm.make_inverse_operator(
    evoked.info, fwd, cov, loose=0., depth=None)
del fwd, cov

method = "MNE"
snr = 3.
lambda2 = 1. / snr ** 2
stc = mne.minimum_norm.apply_inverse(
    evoked, inverse_operator, lambda2, method=method,
    pick_ori=None, verbose=True)

# Plot source estimate at time of best dipole fit
brain = stc.plot(hemi='rh', views='lat', subjects_dir=subjects_dir,
                 initial_time=dip_opm.times[idx],
                 clim=dict(kind='percent', lims=[99, 99.9, 99.99]),
                 size=(400, 300), background='w')


I am working on the file "0.24/_downloads/e7de7baffeeb4beff147bad8657b46dc/opm_data.ipynb".
The first task is:
Prepare data for localization
First we filter and epoch the data:
Can you write Python code for it?

raw = mne.io.read_raw_fif(raw_fname, preload=True)
raw.filter(None, 90, h_trans_bandwidth=10.)
raw.notch_filter(50., notch_widths=1)

# Set epoch rejection threshold a bit larger than for SQUIDs
reject = dict(mag=2e-10)
tmin, tmax = -0.5, 1

# Find median nerve stimulator trigger
event_id = dict(Median=257)
events = mne.find_events(raw, stim_channel='STI101', mask=257, mask_type='and')
picks = mne.pick_types(raw.info, meg=True, eeg=False)
# We use verbose='error' to suppress warning about decimation causing aliasing,
# ideally we would low-pass and then decimate instead
epochs = mne.Epochs(raw, events, event_id, tmin, tmax, verbose='error',
                    reject=reject, picks=picks, proj=False, decim=10,
                    preload=True)
evoked = epochs.average()
evoked.plot()
cov = mne.compute_covariance(epochs, tmax=0.)
del epochs, raw


I am working on the file "0.24/_downloads/ead9220acec394667b95e490359e08e7/70_point_spread.ipynb".
The first task is:
Corrupt known signal with point spread
The aim of this tutorial is to demonstrate how to put a known signal at a
desired location(s) in a :class:mne.SourceEstimate and then corrupt the
signal with point-spread by applying a forward and inverse solution.
Can you write Python code for it?

import os.path as op

import numpy as np

import mne
from mne.datasets import sample

from mne.minimum_norm import read_inverse_operator, apply_inverse
from mne.simulation import simulate_stc, simulate_evoked


I am working on the file "0.24/_downloads/ead9220acec394667b95e490359e08e7/70_point_spread.ipynb".
The first task is:
Create source-space data with known signals
Put known signals onto surface vertices using the array of signals and
the label masks (stored in labels[i].values).
Can you write Python code for it?

stc_gen = simulate_stc(fwd['src'], labels, signal, times[0], dt,
                       value_fun=lambda x: x)


I am working on the file "0.24/_downloads/ead9220acec394667b95e490359e08e7/70_point_spread.ipynb".
The first task is:
Find the center vertices in source space of each label
We want the known signal in each label to only be active at the center. We
create a mask for each label that is 1 at the center vertex and 0 at all
other vertices in the label. This mask is then used when simulating
source-space data.
Can you write Python code for it?

hemi_to_ind = {'lh': 0, 'rh': 1}
for i, label in enumerate(labels):
    # The `center_of_mass` function needs labels to have values.
    labels[i].values.fill(1.)

    # Restrict the eligible vertices to be those on the surface under
    # consideration and within the label.
    surf_vertices = fwd['src'][hemi_to_ind[label.hemi]]['vertno']
    restrict_verts = np.intersect1d(surf_vertices, label.vertices)
    com = labels[i].center_of_mass(subjects_dir=subjects_dir,
                                   restrict_vertices=restrict_verts,
                                   surf='white')

    # Convert the center of vertex index from surface vertex list to Label's
    # vertex list.
    cent_idx = np.where(label.vertices == com)[0][0]

    # Create a mask with 1 at center vertex and zeros elsewhere.
    labels[i].values.fill(0.)
    labels[i].values[cent_idx] = 1.

    # Print some useful information about this vertex and label
    if 'transversetemporal' in label.name:
        dist, _ = label.distances_to_outside(
            subjects_dir=subjects_dir)
        dist = dist[cent_idx]
        area = label.compute_area(subjects_dir=subjects_dir)
        # convert to equivalent circular radius
        r = np.sqrt(area / np.pi)
        print(f'{label.name} COM vertex is {dist * 1e3:0.1f} mm from edge '
              f'(label area equivalent to a circle with r={r * 1e3:0.1f} mm)')


I am working on the file "0.24/_downloads/ead9220acec394667b95e490359e08e7/70_point_spread.ipynb".
The first task is:
First, we set some parameters.
Can you write Python code for it?

seed = 42

# parameters for inverse method
method = 'sLORETA'
snr = 3.
lambda2 = 1.0 / snr ** 2

# signal simulation parameters
# do not add extra noise to the known signals
nave = np.inf
T = 100
times = np.linspace(0, 1, T)
dt = times[1] - times[0]

# Paths to MEG data
data_path = sample.data_path()
subjects_dir = op.join(data_path, 'subjects')
fname_fwd = op.join(data_path, 'MEG', 'sample',
                    'sample_audvis-meg-oct-6-fwd.fif')
fname_inv = op.join(data_path, 'MEG', 'sample',
                    'sample_audvis-meg-oct-6-meg-fixed-inv.fif')

fname_evoked = op.join(data_path, 'MEG', 'sample',
                       'sample_audvis-ave.fif')


I am working on the file "0.24/_downloads/ead9220acec394667b95e490359e08e7/70_point_spread.ipynb".
The first task is:
Generate sinusoids in two spatially distant labels
Can you write Python code for it?

# The known signal is all zero-s off of the two labels of interest
signal = np.zeros((n_labels, T))
idx = label_names.index('inferiorparietal-lh')
signal[idx, :] = 1e-7 * np.sin(5 * 2 * np.pi * times)
idx = label_names.index('rostralmiddlefrontal-rh')
signal[idx, :] = 1e-7 * np.sin(7 * 2 * np.pi * times)


I am working on the file "0.24/_downloads/ead9220acec394667b95e490359e08e7/70_point_spread.ipynb".
The first task is:
Load the MEG data
Can you write Python code for it?

fwd = mne.read_forward_solution(fname_fwd)
fwd = mne.convert_forward_solution(fwd, force_fixed=True, surf_ori=True,
                                   use_cps=False)
fwd['info']['bads'] = []
inv_op = read_inverse_operator(fname_inv)

raw = mne.io.read_raw_fif(op.join(data_path, 'MEG', 'sample',
                                  'sample_audvis_raw.fif'))
raw.set_eeg_reference(projection=True)
events = mne.find_events(raw)
event_id = {'Auditory/Left': 1, 'Auditory/Right': 2}
epochs = mne.Epochs(raw, events, event_id, baseline=(None, 0), preload=True)
epochs.info['bads'] = []
evoked = epochs.average()

labels = mne.read_labels_from_annot('sample', subjects_dir=subjects_dir)
label_names = [label.name for label in labels]
n_labels = len(labels)


I am working on the file "0.24/_downloads/eb0c29f55af0173daab811d4f4dc2f40/simulated_raw_data_using_subject_anatomy.ipynb".
The first task is:
Create simulated source activity
Generate source time courses for each region. In this example, we want to
simulate source activity for a single condition at a time. Therefore, each
evoked response will be parametrized by latency and duration.
Can you write Python code for it?

def data_fun(times, latency, duration):
    """Function to generate source time courses for evoked responses,
    parametrized by latency and duration."""
    f = 15  # oscillating frequency, beta band [Hz]
    sigma = 0.375 * duration
    sinusoid = np.sin(2 * np.pi * f * (times - latency))
    gf = np.exp(- (times - latency - (sigma / 4.) * rng.rand(1)) ** 2 /
                (2 * (sigma ** 2)))
    return 1e-9 * sinusoid * gf


I am working on the file "0.24/_downloads/eb0c29f55af0173daab811d4f4dc2f40/simulated_raw_data_using_subject_anatomy.ipynb".
The first task is:
Extract epochs and compute evoked responsses
Can you write Python code for it?

epochs = mne.Epochs(raw_sim, events, event_id, tmin=-0.2, tmax=0.3,
                    baseline=(None, 0))
evoked_aud_left = epochs['auditory/left'].average()
evoked_vis_right = epochs['visual/right'].average()

# Visualize the evoked data
evoked_aud_left.plot(spatial_colors=True)
evoked_vis_right.plot(spatial_colors=True)


I am working on the file "0.24/_downloads/eb0c29f55af0173daab811d4f4dc2f40/simulated_raw_data_using_subject_anatomy.ipynb".
The first task is:
In order to simulate source time courses, labels of desired active regions
need to be specified for each of the 4 simulation conditions.
Make a dictionary that maps conditions to activation strengths within
aparc.a2009s :footcite:DestrieuxEtAl2010 labels.
In the aparc.a2009s parcellation:

'G_temp_sup-G_T_transv' is the label for primary auditory area
'S_calcarine' is the label for primary visual area

In each of the 4 conditions, only the primary area is activated. This means
that during the activations of auditory areas, there are no activations in
visual areas and vice versa.
Moreover, for each condition, contralateral region is more active (here, 2
times more) than the ipsilateral.
Can you write Python code for it?

activations = {
    'auditory/left':
        [('G_temp_sup-G_T_transv-lh', 30),          # label, activation (nAm)
         ('G_temp_sup-G_T_transv-rh', 60)],
    'auditory/right':
        [('G_temp_sup-G_T_transv-lh', 60),
         ('G_temp_sup-G_T_transv-rh', 30)],
    'visual/left':
        [('S_calcarine-lh', 30),
         ('S_calcarine-rh', 60)],
    'visual/right':
        [('S_calcarine-lh', 60),
         ('S_calcarine-rh', 30)],
}

annot = 'aparc.a2009s'

# Load the 4 necessary label names.
label_names = sorted(set(activation[0]
                         for activation_list in activations.values()
                         for activation in activation_list))
region_names = list(activations.keys())


I am working on the file "0.24/_downloads/eb0c29f55af0173daab811d4f4dc2f40/simulated_raw_data_using_subject_anatomy.ipynb".
The first task is:
Reconstruct simulated source time courses using dSPM inverse operator
Here, source time courses for auditory and visual areas are reconstructed
separately and their difference is shown. This was done merely for better
visual representation of source reconstruction.
As expected, when high activations appear in primary auditory areas, primary
visual areas will have low activations and vice versa.
Can you write Python code for it?

method, lambda2 = 'dSPM', 1. / 9.
inv = mne.minimum_norm.make_inverse_operator(epochs.info, fwd, noise_cov)
stc_aud = mne.minimum_norm.apply_inverse(
    evoked_aud_left, inv, lambda2, method)
stc_vis = mne.minimum_norm.apply_inverse(
    evoked_vis_right, inv, lambda2, method)
stc_diff = stc_aud - stc_vis

brain = stc_diff.plot(subjects_dir=subjects_dir, initial_time=0.1,
                      hemi='split', views=['lat', 'med'])


I am working on the file "0.24/_downloads/eb0c29f55af0173daab811d4f4dc2f40/simulated_raw_data_using_subject_anatomy.ipynb".
The first task is:
Simulate raw data
Project the source time series to sensor space. Three types of noise will be
added to the simulated raw data:

multivariate Gaussian noise obtained from the noise covariance from the
  sample data
blink (EOG) noise
ECG noise

The :class:~mne.simulation.SourceSimulator can be given directly to the
:func:~mne.simulation.simulate_raw function.
Can you write Python code for it?

raw_sim = mne.simulation.simulate_raw(info, source_simulator, forward=fwd)
raw_sim.set_eeg_reference(projection=True)

mne.simulation.add_noise(raw_sim, cov=noise_cov, random_state=0)
mne.simulation.add_eog(raw_sim, random_state=0)
mne.simulation.add_ecg(raw_sim, random_state=0)

# Plot original and simulated raw data.
raw_sim.plot(title='Simulated raw data')


I am working on the file "0.24/_downloads/eb0c29f55af0173daab811d4f4dc2f40/simulated_raw_data_using_subject_anatomy.ipynb".
The first task is:
Simulate raw data using subject anatomy
This example illustrates how to generate source estimates and simulate raw data
using subject anatomy with the :class:mne.simulation.SourceSimulator class.
Once the raw data is simulated, generated source estimates are reconstructed
using dynamic statistical parametric mapping (dSPM) inverse operator.
Can you write Python code for it?

# Author: Ivana Kojcic <ivana.kojcic@gmail.com>
#         Eric Larson <larson.eric.d@gmail.com>
#         Kostiantyn Maksymenko <kostiantyn.maksymenko@gmail.com>
#         Samuel Deslauriers-Gauthier <sam.deslauriers@gmail.com>

# License: BSD-3-Clause

import os.path as op

import numpy as np

import mne
from mne.datasets import sample

print(__doc__)

# In this example, raw data will be simulated for the sample subject, so its
# information needs to be loaded. This step will download the data if it not
# already on your machine. Subjects directory is also set so it doesn't need
# to be given to functions.
data_path = sample.data_path()
subjects_dir = op.join(data_path, 'subjects')
subject = 'sample'
meg_path = op.join(data_path, 'MEG', subject)

# First, we get an info structure from the sample subject.
fname_info = op.join(meg_path, 'sample_audvis_raw.fif')
info = mne.io.read_info(fname_info)
tstep = 1 / info['sfreq']

# To simulate sources, we also need a source space. It can be obtained from the
# forward solution of the sample subject.
fwd_fname = op.join(meg_path, 'sample_audvis-meg-eeg-oct-6-fwd.fif')
fwd = mne.read_forward_solution(fwd_fname)
src = fwd['src']

# To simulate raw data, we need to define when the activity occurs using events
# matrix and specify the IDs of each event.
# Noise covariance matrix also needs to be defined.
# Here, both are loaded from the sample dataset, but they can also be specified
# by the user.

fname_event = op.join(meg_path, 'sample_audvis_raw-eve.fif')
fname_cov = op.join(meg_path, 'sample_audvis-cov.fif')

events = mne.read_events(fname_event)
noise_cov = mne.read_cov(fname_cov)

# Standard sample event IDs. These values will correspond to the third column
# in the events matrix.
event_id = {'auditory/left': 1, 'auditory/right': 2, 'visual/left': 3,
            'visual/right': 4, 'smiley': 5, 'button': 32}

# Take only a few events for speed
events = events[:80]


I am working on the file "0.24/_downloads/ed1a04dd775648ca869bfcffae26faca/30_mne_dspm_loreta.ipynb".
The first task is:
Examine the original data and the residual after fitting:
Can you write Python code for it?

fig, axes = plt.subplots(2, 1)
evoked.plot(axes=axes)
for ax in axes:
    for text in list(ax.texts):
        text.remove()
    for line in ax.lines:
        line.set_color('#98df81')
residual.plot(axes=axes)


I am working on the file "0.24/_downloads/ed1a04dd775648ca869bfcffae26faca/30_mne_dspm_loreta.ipynb".
The first task is:
Here we use peak getter to move visualization to the time point of the peak
and draw a marker at the maximum peak vertex.
Can you write Python code for it?

vertno_max, time_max = stc.get_peak(hemi='rh')

subjects_dir = data_path + '/subjects'
surfer_kwargs = dict(
    hemi='rh', subjects_dir=subjects_dir,
    clim=dict(kind='value', lims=[8, 12, 15]), views='lateral',
    initial_time=time_max, time_unit='s', size=(800, 800), smoothing_steps=10)
brain = stc.plot(**surfer_kwargs)
brain.add_foci(vertno_max, coords_as_verts=True, hemi='rh', color='blue',
               scale_factor=0.6, alpha=0.5)
brain.add_text(0.1, 0.9, 'dSPM (plus location of maximal activation)', 'title',
               font_size=14)

# The documentation website's movie is generated with:
# brain.save_movie(..., tmin=0.05, tmax=0.15, interpolation='linear',
#                  time_dilation=20, framerate=10, time_viewer=True)


I am working on the file "0.24/_downloads/ed1a04dd775648ca869bfcffae26faca/30_mne_dspm_loreta.ipynb".
The first task is:
Inverse modeling: MNE/dSPM on evoked and raw data
Here we first read the forward solution. You will likely need to compute
one for your own data -- see tut-forward for information on how
to do it.
Can you write Python code for it?

fname_fwd = data_path + '/MEG/sample/sample_audvis-meg-oct-6-fwd.fif'
fwd = mne.read_forward_solution(fname_fwd)


I am working on the file "0.24/_downloads/ed1a04dd775648ca869bfcffae26faca/30_mne_dspm_loreta.ipynb".
The first task is:
Visualization
We can look at different dipole activations:
Can you write Python code for it?

fig, ax = plt.subplots()
ax.plot(1e3 * stc.times, stc.data[::100, :].T)
ax.set(xlabel='time (ms)', ylabel='%s value' % method)


I am working on the file "0.24/_downloads/efd09079125b2bd222e2dd62aaaccfa4/source_space_snr.ipynb".
The first task is:
Computing source space SNR
This example shows how to compute and plot source space SNR as in
:footcite:GoldenholzEtAl2009.
Can you write Python code for it?

# Author: Padma Sundaram <tottochan@gmail.com>
#         Kaisu Lankinen <klankinen@mgh.harvard.edu>
#
# License: BSD-3-Clause

import mne
from mne.datasets import sample
from mne.minimum_norm import make_inverse_operator, apply_inverse
import numpy as np
import matplotlib.pyplot as plt

print(__doc__)

data_path = sample.data_path()
subjects_dir = data_path + '/subjects'

# Read data
fname_evoked = data_path + '/MEG/sample/sample_audvis-ave.fif'
evoked = mne.read_evokeds(fname_evoked, condition='Left Auditory',
                          baseline=(None, 0))
fname_fwd = data_path + '/MEG/sample/sample_audvis-meg-eeg-oct-6-fwd.fif'
fname_cov = data_path + '/MEG/sample/sample_audvis-cov.fif'
fwd = mne.read_forward_solution(fname_fwd)
cov = mne.read_cov(fname_cov)

# Read inverse operator:
inv_op = make_inverse_operator(evoked.info, fwd, cov, fixed=True, verbose=True)

# Calculate MNE:
snr = 3.0
lambda2 = 1.0 / snr ** 2
stc = apply_inverse(evoked, inv_op, lambda2, 'MNE', verbose=True)

# Calculate SNR in source space:
snr_stc = stc.estimate_snr(evoked.info, fwd, cov)

# Plot an average SNR across source points over time:
ave = np.mean(snr_stc.data, axis=0)

fig, ax = plt.subplots()
ax.plot(evoked.times, ave)
ax.set(xlabel='Time (sec)', ylabel='SNR MEG-EEG')
fig.tight_layout()

# Find time point of maximum SNR
maxidx = np.argmax(ave)

# Plot SNR on source space at the time point of maximum SNR:
kwargs = dict(initial_time=evoked.times[maxidx], hemi='split',
              views=['lat', 'med'], subjects_dir=subjects_dir, size=(600, 600),
              clim=dict(kind='value', lims=(-100, -70, -40)),
              transparent=True, colormap='viridis')
brain = snr_stc.plot(**kwargs)


I am working on the file "0.24/_downloads/efd09079125b2bd222e2dd62aaaccfa4/source_space_snr.ipynb".
The first task is:
EEG
Next we do the same for EEG and plot the result on the cortex:
Can you write Python code for it?

evoked_eeg = evoked.copy().pick_types(eeg=True, meg=False)
inv_op_eeg = make_inverse_operator(evoked_eeg.info, fwd, cov, fixed=True,
                                   verbose=True)
stc_eeg = apply_inverse(evoked_eeg, inv_op_eeg, lambda2, 'MNE', verbose=True)
snr_stc_eeg = stc_eeg.estimate_snr(evoked_eeg.info, fwd, cov)
brain = snr_stc_eeg.plot(**kwargs)


I am working on the file "0.24/_downloads/f1d68aba13226287585e777005a39f0a/15_handling_bad_channels.ipynb".
The first task is:
.. sidebar:: Blocking execution
If you want to build an interactive bad-channel-marking step into an
analysis script, be sure to include the parameter ``block=True`` in your
call to ``raw.plot()`` or ``epochs.plot()``. This will pause the script
while the plot is open, giving you time to mark bad channels before
subsequent analysis or plotting steps are executed. This can be
especially helpful if your script loops over multiple subjects.

You can also interactively toggle whether a channel is marked "bad" in the
plot windows of raw.plot() or epochs.plot() by clicking on the
channel name along the vertical axis (in raw.plot() windows you can also
do this by clicking the channel's trace in the plot area). The bads field
gets updated immediately each time you toggle a channel, and will retain its
modified state after the plot window is closed.
The list of bad channels in the :class:mne.Info object's bads field is
automatically taken into account in dozens of functions and methods across
the MNE-Python codebase. This is done consistently with a parameter
exclude='bads' in the function or method signature. Typically this
exclude parameter also accepts a list of channel names or indices, so if
you want to include the bad channels you can do so by passing
exclude=[] (or some other list of channels to exclude). For example:
Can you write Python code for it?

# default is exclude='bads':
good_eeg = mne.pick_types(raw.info, meg=False, eeg=True)
all_eeg = mne.pick_types(raw.info, meg=False, eeg=True, exclude=[])
print(np.setdiff1d(all_eeg, good_eeg))
print(np.array(raw.ch_names)[np.setdiff1d(all_eeg, good_eeg)])


I am working on the file "0.24/_downloads/f1d68aba13226287585e777005a39f0a/15_handling_bad_channels.ipynb".
The first task is:
Note that we used the exclude=[] trick in the call to
:meth:~mne.io.Raw.pick_types to make sure the bad channels were not
automatically dropped from the selection. Here is the corresponding example
with the interpolated gradiometer channel; since there are more channels
we'll use a more transparent gray color this time:
Can you write Python code for it?

grad_data = raw.copy().pick_types(meg='grad', exclude=[])
grad_data_interp = grad_data.copy().interpolate_bads(reset_bads=False)

for data in (grad_data, grad_data_interp):
    data.plot(butterfly=True, color='#00000009', bad_color='r')


I am working on the file "0.24/_downloads/f1d68aba13226287585e777005a39f0a/15_handling_bad_channels.ipynb".
The first task is:
We can do the same thing for the bad MEG channel (MEG 2443). Since we
know that Neuromag systems (like the one used to record the example data) use
the last digit of the MEG channel number to indicate sensor type, here our
regular expression_ will pick all the channels that start with 2 and end
with 3:
Can you write Python code for it?

picks = mne.pick_channels_regexp(raw.ch_names, regexp='MEG 2..3')
raw.plot(order=picks, n_channels=len(picks))


I am working on the file "0.24/_downloads/f1d68aba13226287585e777005a39f0a/15_handling_bad_channels.ipynb".
The first task is:
When to look for bad channels
You can start looking for bad channels during the experiment session when the
data is being acquired. If you notice any flat or excessively noisy channels,
you can note them in your experiment log or protocol sheet. If your system
computes online averages, these can be a good way to spot bad channels as
well. After the data has been collected, you can do a more thorough check for
bad channels by browsing the raw data using :meth:mne.io.Raw.plot, without
any projectors or ICA applied. Finally, you can compute offline averages
(again with projectors, ICA, and EEG referencing disabled) to look for
channels with unusual properties. Here's an example of ERP/F plots where the
bad channels were not properly marked:
Can you write Python code for it?

raw2 = raw.copy()
raw2.info['bads'] = []
events = mne.find_events(raw2, stim_channel='STI 014')
epochs = mne.Epochs(raw2, events=events)['2'].average().plot()


I am working on the file "0.24/_downloads/f31e73ee907864d95a2b617fdc76b71e/source_label_time_frequency.ipynb".
The first task is:
Compute power and phase lock in label of the source space
Compute time-frequency maps of power and phase lock in the source space.
The inverse method is linear based on dSPM inverse operator.
The example also shows the difference in the time-frequency maps
when they are computed with and without subtracting the evoked response
from each epoch. The former results in induced activity only while the
latter also includes evoked (stimulus-locked) activity.
Can you write Python code for it?

# Authors: Alexandre Gramfort <alexandre.gramfort@inria.fr>
#
# License: BSD-3-Clause

import numpy as np
import matplotlib.pyplot as plt

import mne
from mne import io
from mne.datasets import sample
from mne.minimum_norm import read_inverse_operator, source_induced_power

print(__doc__)


I am working on the file "0.24/_downloads/f398f296c84e53a14339d2c3c36e91a4/movement_detection.ipynb".
The first task is:
After checking the annotated movement artifacts, calculate the new transform
and plot it:
Can you write Python code for it?

new_dev_head_t = compute_average_dev_head_t(raw, head_pos)
raw.info['dev_head_t'] = new_dev_head_t
mne.viz.plot_alignment(raw.info, show_axes=True, subject=subject,
                       trans=trans_fname, subjects_dir=subjects_dir)


I am working on the file "0.24/_downloads/f398f296c84e53a14339d2c3c36e91a4/movement_detection.ipynb".
The first task is:
Annotate movement artifacts and reestimate dev_head_t
Periods, where the participant moved considerably, are contaminated by low
amplitude artifacts. When averaging the magnetic fields, the more spread the
head position, the bigger the cancellation due to different locations.
Similarly, the covariance will also be affected by severe head movement,
and source estimation will suffer low/smeared coregistration accuracy.
This example uses the continuous head position indicators (cHPI) times series
to annotate periods of head movement, then the device to head transformation
matrix is estimated from the artifact-free segments. The new head position will
be more representative of the actual head position during the recording.
Can you write Python code for it?

# Authors: Adonay Nunes <adonay.s.nunes@gmail.com>
#          Luke Bloy <luke.bloy@gmail.com>
# License: BSD-3-Clause

import os.path as op

import mne
from mne.datasets.brainstorm import bst_auditory
from mne.io import read_raw_ctf
from mne.preprocessing import annotate_movement, compute_average_dev_head_t

# Load data
data_path = bst_auditory.data_path()
data_path_MEG = op.join(data_path, 'MEG')
subject = 'bst_auditory'
subjects_dir = op.join(data_path, 'subjects')
trans_fname = op.join(data_path, 'MEG', 'bst_auditory',
                      'bst_auditory-trans.fif')
raw_fname1 = op.join(data_path_MEG, 'bst_auditory', 'S01_AEF_20131218_01.ds')
raw_fname2 = op.join(data_path_MEG, 'bst_auditory', 'S01_AEF_20131218_02.ds')
# read and concatenate two files, ignoring device<->head mismatch
raw = read_raw_ctf(raw_fname1, preload=False)
mne.io.concatenate_raws(
    [raw, read_raw_ctf(raw_fname2, preload=False)], on_mismatch='ignore')
raw.crop(350, 410).load_data()
raw.resample(100, npad="auto")


I am working on the file "0.24/_downloads/f398f296c84e53a14339d2c3c36e91a4/movement_detection.ipynb".
The first task is:
Plot continuous head position with respect to the mean recording position
Can you write Python code for it?

# Get cHPI time series and compute average
chpi_locs = mne.chpi.extract_chpi_locs_ctf(raw)
head_pos = mne.chpi.compute_head_pos(raw.info, chpi_locs)
original_head_dev_t = mne.transforms.invert_transform(
    raw.info['dev_head_t'])
average_head_dev_t = mne.transforms.invert_transform(
    compute_average_dev_head_t(raw, head_pos))
fig = mne.viz.plot_head_positions(head_pos)
for ax, val, val_ori in zip(fig.axes[::2], average_head_dev_t['trans'][:3, 3],
                            original_head_dev_t['trans'][:3, 3]):
    ax.axhline(1000 * val, color='r')
    ax.axhline(1000 * val_ori, color='g')

# The green horizontal lines represent the original head position, whereas the
# red lines are the new head position averaged over all the time points.


I am working on the file "0.24/_downloads/f398f296c84e53a14339d2c3c36e91a4/movement_detection.ipynb".
The first task is:
Plot raw data with annotated movement
Can you write Python code for it?

mean_distance_limit = .0015  # in meters
annotation_movement, hpi_disp = annotate_movement(
    raw, head_pos, mean_distance_limit=mean_distance_limit)
raw.set_annotations(annotation_movement)
raw.plot(n_channels=100, duration=20)


I am working on the file "0.24/_downloads/f574d1e7527e4460eb09a16f6f836e35/60_maxwell_filtering_sss.ipynb".
The first task is:

Note

You can use the very same code as above to produce figures for
          *flat* channel detection. Simply replace the word "noisy" with
          "flat", and replace ``vmin=np.nanmin(limits)`` with
          ``vmax=np.nanmax(limits)``.

You can see the un-altered scores for each channel and time segment in the
left subplots, and thresholded scores – those which exceeded a certain limit
of noisiness – in the right subplots. While the right subplot is entirely
white for the magnetometers, we can see a horizontal line extending all the
way from left to right for the gradiometers. This line corresponds to channel
MEG 2443, which was reported as auto-detected noisy channel in the step
above. But we can also see another channel exceeding the limits, apparently
in a more transient fashion. It was therefore not detected as bad, because
the number of segments in which it exceeded the limits was less than 5,
which MNE-Python uses by default.

Note

You can request a different number of segments that must be
          found to be problematic before
          `~mne.preprocessing.find_bad_channels_maxwell` reports them as bad.
          To do this, pass the keyword argument ``min_count`` to the
          function.

Obviously, this algorithm is not perfect. Specifically, on closer inspection
of the raw data after looking at the diagnostic plots above, it becomes clear
that the channel exceeding the "noise" limits in some segments without
qualifying as "bad", in fact contains some flux jumps. There were just not
enough flux jumps in the recording for our automated procedure to report
the channel as bad. So it can still be useful to manually inspect and mark
bad channels. The channel in question is MEG 2313. Let's mark it as bad:
Can you write Python code for it?

raw.info['bads'] += ['MEG 2313']  # from manual inspection


I am working on the file "0.24/_downloads/f574d1e7527e4460eb09a16f6f836e35/60_maxwell_filtering_sss.ipynb".
The first task is:

Note

`~mne.preprocessing.find_bad_channels_maxwell` needs to operate on
          a signal without line noise or cHPI signals. By default, it simply
          applies a low-pass filter with a cutoff frequency of 40 Hz to the
          data, which should remove these artifacts. You may also specify a
          different cutoff by passing the ``h_freq`` keyword argument. If you
          set ``h_freq=None``, no filtering will be applied. This can be
          useful if your data has already been preconditioned, for example
          using :func:`mne.chpi.filter_chpi`,
          :func:`mne.io.Raw.notch_filter`, or :meth:`mne.io.Raw.filter`.

Now we can update the list of bad channels in the dataset.
Can you write Python code for it?

bads = raw.info['bads'] + auto_noisy_chs + auto_flat_chs
raw.info['bads'] = bads


I am working on the file "0.24/_downloads/f574d1e7527e4460eb09a16f6f836e35/60_maxwell_filtering_sss.ipynb".
The first task is:
Background on SSS and Maxwell filtering
Signal-space separation (SSS) :footcite:TauluKajola2005,TauluSimola2006
is a technique based on the physics
of electromagnetic fields. SSS separates the measured signal into components
attributable to sources inside the measurement volume of the sensor array
(the internal components), and components attributable to sources outside
the measurement volume (the external components). The internal and external
components are linearly independent, so it is possible to simply discard the
external components to reduce environmental noise. Maxwell filtering is a
related procedure that omits the higher-order components of the internal
subspace, which are dominated by sensor noise. Typically, Maxwell filtering
and SSS are performed together (in MNE-Python they are implemented together
in a single function).
Like SSP <tut-artifact-ssp>, SSS is a form of projection. Whereas SSP
empirically determines a noise subspace based on data (empty-room recordings,
EOG or ECG activity, etc) and projects the measurements onto a subspace
orthogonal to the noise, SSS mathematically constructs the external and
internal subspaces from spherical harmonics_ and reconstructs the sensor
signals using only the internal subspace (i.e., does an oblique projection).

Warning

Maxwell filtering was originally developed for Elekta Neuromag® systems,
    and should be considered *experimental* for non-Neuromag data. See the
    Notes section of the :func:`~mne.preprocessing.maxwell_filter` docstring
    for details.

The MNE-Python implementation of SSS / Maxwell filtering currently provides
the following features:

Basic bad channel detection
  (:func:~mne.preprocessing.find_bad_channels_maxwell)
Bad channel reconstruction
Cross-talk cancellation
Fine calibration correction
tSSS
Coordinate frame translation
Regularization of internal components using information theory
Raw movement compensation (using head positions estimated by MaxFilter)
cHPI...
Can you write Python code for it?

fine_cal_file = os.path.join(sample_data_folder, 'SSS', 'sss_cal_mgh.dat')
crosstalk_file = os.path.join(sample_data_folder, 'SSS', 'ct_sparse_mgh.fif')


I am working on the file "0.24/_downloads/f574d1e7527e4460eb09a16f6f836e35/60_maxwell_filtering_sss.ipynb".
The first task is:
Notice that channels marked as "bad" have been effectively repaired by SSS,
eliminating the need to perform interpolation <tut-bad-channels>.
The heartbeat artifact has also been substantially reduced.
The :func:~mne.preprocessing.maxwell_filter function has parameters
int_order and ext_order for setting the order of the spherical
harmonic expansion of the interior and exterior components; the default
values are appropriate for most use cases. Additional parameters include
coord_frame and origin for controlling the coordinate frame ("head"
or "meg") and the origin of the sphere; the defaults are appropriate for most
studies that include digitization of the scalp surface / electrodes. See the
documentation of :func:~mne.preprocessing.maxwell_filter for details.
Spatiotemporal SSS (tSSS)
An assumption of SSS is that the measurement volume (the spherical shell
where the sensors are physically located) is free of electromagnetic sources.
The thickness of this source-free measurement shell should be 4-8 cm for SSS
to perform optimally. In practice, there may be sources falling within that
measurement volume; these can often be mitigated by using Spatiotemporal
Signal Space Separation (tSSS) :footcite:TauluSimola2006.
tSSS works by looking for temporal
correlation between components of the internal and external subspaces, and
projecting out any components that are common to the internal and external
subspaces. The projection is done in an analogous way to
SSP <tut-artifact-ssp>, except that the noise vector is computed
across time points instead of across sensors.
To use tSSS in MNE-Python, pass a time (in seconds) to the parameter
st_duration of :func:~mne.preprocessing.maxwell_filter. This will
determine the "chunk duration" over which to compute the temporal projection.
The chunk duration effectively acts as a high-pass filter with a cutoff
frequency of $\frac{1}{\mathtt{st_duration}}~\mathrm{Hz}$; this
effective high-pass has an important consequence:

In general, larger values of st_duration are...
Can you write Python code for it?

head_pos_file = os.path.join(mne.datasets.testing.data_path(), 'SSS',
                             'test_move_anon_raw.pos')
head_pos = mne.chpi.read_head_pos(head_pos_file)
mne.viz.plot_head_positions(head_pos, mode='traces')


I am working on the file "0.24/_downloads/f574d1e7527e4460eb09a16f6f836e35/60_maxwell_filtering_sss.ipynb".
The first task is:
Signal-space separation (SSS) and Maxwell filtering
This tutorial covers reducing environmental noise and compensating for head
movement with SSS and Maxwell filtering.
As usual we'll start by importing the modules we need, loading some
example data <sample-dataset>, and cropping it to save on memory:
Can you write Python code for it?

import os
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import numpy as np
import mne
from mne.preprocessing import find_bad_channels_maxwell

sample_data_folder = mne.datasets.sample.data_path()
sample_data_raw_file = os.path.join(sample_data_folder, 'MEG', 'sample',
                                    'sample_audvis_raw.fif')
raw = mne.io.read_raw_fif(sample_data_raw_file, verbose=False)
raw.crop(tmax=60)


I am working on the file "0.24/_downloads/f574d1e7527e4460eb09a16f6f836e35/60_maxwell_filtering_sss.ipynb".
The first task is:
To see the effect, we can plot the data before and after SSS / Maxwell
filtering.
Can you write Python code for it?

raw.pick(['meg']).plot(duration=2, butterfly=True)
raw_sss.pick(['meg']).plot(duration=2, butterfly=True)


I am working on the file "0.24/_downloads/f574d1e7527e4460eb09a16f6f836e35/60_maxwell_filtering_sss.ipynb".
The first task is:
We called ~mne.preprocessing.find_bad_channels_maxwell with the optional
keyword argument return_scores=True, causing the function to return a
dictionary of all data related to the scoring used to classify channels as
noisy or flat. This information can be used to produce diagnostic figures.
In the following, we will generate such visualizations for
the automated detection of noisy gradiometer channels.
Can you write Python code for it?

# Only select the data forgradiometer channels.
ch_type = 'grad'
ch_subset = auto_scores['ch_types'] == ch_type
ch_names = auto_scores['ch_names'][ch_subset]
scores = auto_scores['scores_noisy'][ch_subset]
limits = auto_scores['limits_noisy'][ch_subset]
bins = auto_scores['bins']  # The the windows that were evaluated.
# We will label each segment by its start and stop time, with up to 3
# digits before and 3 digits after the decimal place (1 ms precision).
bin_labels = [f'{start:3.3f} – {stop:3.3f}'
              for start, stop in bins]

# We store the data in a Pandas DataFrame. The seaborn heatmap function
# we will call below will then be able to automatically assign the correct
# labels to all axes.
data_to_plot = pd.DataFrame(data=scores,
                            columns=pd.Index(bin_labels, name='Time (s)'),
                            index=pd.Index(ch_names, name='Channel'))

# First, plot the "raw" scores.
fig, ax = plt.subplots(1, 2, figsize=(12, 8))
fig.suptitle(f'Automated noisy channel detection: {ch_type}',
             fontsize=16, fontweight='bold')
sns.heatmap(data=data_to_plot, cmap='Reds', cbar_kws=dict(label='Score'),
            ax=ax[0])
[ax[0].axvline(x, ls='dashed', lw=0.25, dashes=(25, 15), color='gray')
    for x in range(1, len(bins))]
ax[0].set_title('All Scores', fontweight='bold')

# Now, adjust the color range to highlight segments that exceeded the limit.
sns.heatmap(data=data_to_plot,
            vmin=np.nanmin(limits),  # bads in input data have NaN limits
            cmap='Reds', cbar_kws=dict(label='Score'), ax=ax[1])
[ax[1].axvline(x, ls='dashed', lw=0.25, dashes=(25, 15), color='gray')
    for x in range(1, len(bins))]
ax[1].set_title('Scores > Limit', fontweight='bold')

# The figure title should not overlap with the subplots.
fig.tight_layout(rect=[0, 0.03, 1, 0.95])


I am working on the file "0.24/_downloads/f5853db1ea98f82173310d147f23289c/compute_mne_inverse_epochs_in_label.ipynb".
The first task is:
View activation time-series to illustrate the benefit of aligning/flipping
Can you write Python code for it?

times = 1e3 * stcs[0].times  # times in ms

plt.figure()
h0 = plt.plot(times, mean_stc.data.T, 'k')
h1, = plt.plot(times, label_mean, 'r', linewidth=3)
h2, = plt.plot(times, label_mean_flip, 'g', linewidth=3)
plt.legend((h0[0], h1, h2), ('all dipoles in label', 'mean',
                             'mean with sign flip'))
plt.xlabel('time (ms)')
plt.ylabel('dSPM value')
plt.show()


I am working on the file "0.24/_downloads/f5853db1ea98f82173310d147f23289c/compute_mne_inverse_epochs_in_label.ipynb".
The first task is:
Viewing single trial dSPM and average dSPM for unflipped pooling over label
Compare to (1) Inverse (dSPM) then average, (2) Evoked then dSPM
Can you write Python code for it?

# Single trial
plt.figure()
for k, stc_trial in enumerate(stcs):
    plt.plot(times, np.mean(stc_trial.data, axis=0).T, 'k--',
             label='Single Trials' if k == 0 else '_nolegend_',
             alpha=0.5)

# Single trial inverse then average.. making linewidth large to not be masked
plt.plot(times, label_mean, 'b', linewidth=6,
         label='dSPM first, then average')

# Evoked and then inverse
plt.plot(times, label_mean_evoked, 'r', linewidth=2,
         label='Average first, then dSPM')

plt.xlabel('time (ms)')
plt.ylabel('dSPM value')
plt.legend()
plt.show()


I am working on the file "0.24/_downloads/fb92190904499e5a95e92ab70177abf7/60_make_fixed_length_epochs.ipynb".
The first task is:
Characteristics of Fixed Length Epochs
Fixed length epochs are generally unsuitable for event-related analyses. This
can be seen in an image map of our fixed length
epochs. When the epochs are averaged, as seen at the bottom of the plot,
misalignment between onsets of event-related activity results in noise.
Can you write Python code for it?

event_related_plot = epochs.plot_image(picks=['MEG 1142'])


I am working on the file "0.24/_downloads/fb92190904499e5a95e92ab70177abf7/60_make_fixed_length_epochs.ipynb".
The first task is:
Divide continuous data into equally-spaced epochs
This tutorial shows how to segment continuous data into a set of epochs spaced
equidistantly in time. The epochs will not be created based on experimental
events; instead, the continuous data will be "chunked" into consecutive epochs
(which may be temporally overlapping, adjacent, or separated).
We will also briefly demonstrate how to use these epochs in connectivity
analysis.
First, we import necessary modules and read in a sample raw data set.
This data set contains brain activity that is event-related, i.e.,
synchronized to the onset of auditory stimuli. However, rather than creating
epochs by segmenting the data around the onset of each stimulus, we will
create 30 second epochs that allow us to perform non-event-related analyses of
the signal.

Note

Starting in version 0.25, all functions in the ``mne.connectivity``
    sub-module will be housed in a separate package called
    :mod:`mne-connectivity <mne_connectivity>`. Download it by  running:

    .. code-block:: console

        $ pip install mne-connectivity

Can you write Python code for it?

import os
import numpy as np
import matplotlib.pyplot as plt
import mne
from mne.preprocessing import compute_proj_ecg
from mne_connectivity import envelope_correlation

sample_data_folder = mne.datasets.sample.data_path()
sample_data_raw_file = os.path.join(sample_data_folder, 'MEG', 'sample',
                                    'sample_audvis_raw.fif')

raw = mne.io.read_raw_fif(sample_data_raw_file)


I am working on the file "0.24/_downloads/fb92190904499e5a95e92ab70177abf7/60_make_fixed_length_epochs.ipynb".
The first task is:
For information about creating epochs for event-related analyses, please see
tut-epochs-class.
Example Use Case for Fixed Length Epochs: Connectivity Analysis
Fixed lengths epochs are suitable for many types of analysis, including
frequency or time-frequency analyses, connectivity analyses, or
classification analyses. Here we briefly illustrate their utility in a sensor
space connectivity analysis.
The data from our epochs object has shape (n_epochs, n_sensors, n_times)
and is therefore an appropriate basis for using MNE-Python's envelope
correlation function to compute power-based connectivity in sensor space. The
long duration of our fixed length epochs, 30 seconds, helps us reduce edge
artifacts and achieve better frequency resolution when filtering must
be applied after epoching.
Let's examine the alpha band. We allow default values for filter parameters
(for more information on filtering, please see tut-filter-resample).
Can you write Python code for it?

epochs.load_data().filter(l_freq=8, h_freq=12)
alpha_data = epochs.get_data()


I am working on the file "0.24/_downloads/fb92190904499e5a95e92ab70177abf7/60_make_fixed_length_epochs.ipynb".
The first task is:
To create fixed length epochs, we simply call the function and provide it
with the appropriate parameters indicating the desired duration of epochs in
seconds, whether or not to preload data, whether or not to reject epochs that
overlap with raw data segments annotated as bad, whether or not to include
projectors, and finally whether or not to be verbose. Here, we choose a long
epoch duration (30 seconds). To conserve memory, we set preload to
False.
Can you write Python code for it?

epochs = mne.make_fixed_length_epochs(raw, duration=30, preload=False)


I am working on the file "0.24/_downloads/fcc5782db3e2930fc79f31bc745495ed/60_ctf_bst_auditory.ipynb".
The first task is:
Difference.
Can you write Python code for it?

stc_difference = apply_inverse(evoked_difference, inv, lambda2, 'dSPM')
brain = stc_difference.plot(subjects_dir=subjects_dir, subject=subject,
                            surface='inflated', time_viewer=False, hemi='lh',
                            initial_time=0.15, time_unit='s')


I am working on the file "0.24/_downloads/fcc5782db3e2930fc79f31bc745495ed/60_ctf_bst_auditory.ipynb".
The first task is:
Epoching and averaging.
First some parameters are defined and events extracted from the stimulus
channel (UPPT001). The rejection thresholds are defined as peak-to-peak
values and are in T / m for gradiometers, T for magnetometers and
V for EOG and EEG channels.
Can you write Python code for it?

tmin, tmax = -0.1, 0.5
event_id = dict(standard=1, deviant=2)
reject = dict(mag=4e-12, eog=250e-6)
# find events
events = mne.find_events(raw, stim_channel='UPPT001')


I am working on the file "0.24/_downloads/fcc5782db3e2930fc79f31bc745495ed/60_ctf_bst_auditory.ipynb".
The first task is:
For noise reduction, a set of bad segments have been identified and stored
in csv files. The bad segments are later used to reject epochs that overlap
with them.
The file for the second run also contains some saccades. The saccades are
removed by using SSP. We use pandas to read the data from the csv files. You
can also view the files with your favorite text editor.
Can you write Python code for it?

annotations_df = pd.DataFrame()
offset = n_times_run1
for idx in [1, 2]:
    csv_fname = op.join(data_path, 'MEG', 'bst_auditory',
                        'events_bad_0%s.csv' % idx)
    df = pd.read_csv(csv_fname, header=None,
                     names=['onset', 'duration', 'id', 'label'])
    print('Events from run {0}:'.format(idx))
    print(df)

    df['onset'] += offset * (idx - 1)
    annotations_df = pd.concat([annotations_df, df], axis=0)

saccades_events = df[df['label'] == 'saccade'].values[:, :3].astype(int)

# Conversion from samples to times:
onsets = annotations_df['onset'].values / raw.info['sfreq']
durations = annotations_df['duration'].values / raw.info['sfreq']
descriptions = annotations_df['label'].values

annotations = mne.Annotations(onsets, durations, descriptions)
raw.set_annotations(annotations)
del onsets, durations, descriptions


I am working on the file "0.24/_downloads/fcc5782db3e2930fc79f31bc745495ed/60_ctf_bst_auditory.ipynb".
The first task is:
Here we compute the saccade and EOG projectors for magnetometers and add
them to the raw data. The projectors are added to both runs.
Can you write Python code for it?

saccade_epochs = mne.Epochs(raw, saccades_events, 1, 0., 0.5, preload=True,
                            baseline=(None, None),
                            reject_by_annotation=False)

projs_saccade = mne.compute_proj_epochs(saccade_epochs, n_mag=1, n_eeg=0,
                                        desc_prefix='saccade')
if use_precomputed:
    proj_fname = op.join(data_path, 'MEG', 'bst_auditory',
                         'bst_auditory-eog-proj.fif')
    projs_eog = mne.read_proj(proj_fname)[0]
else:
    projs_eog, _ = mne.preprocessing.compute_proj_eog(raw.load_data(),
                                                      n_mag=1, n_eeg=0)
raw.add_proj(projs_saccade)
raw.add_proj(projs_eog)
del saccade_epochs, saccades_events, projs_eog, projs_saccade  # To save memory


I am working on the file "0.24/_downloads/fcc5782db3e2930fc79f31bc745495ed/60_ctf_bst_auditory.ipynb".
The first task is:
In the memory saving mode we use preload=False and use the memory
efficient IO which loads the data on demand. However, filtering and some
other functions require the data to be preloaded into memory.
Can you write Python code for it?

raw = read_raw_ctf(raw_fname1)
n_times_run1 = raw.n_times

# Here we ignore that these have different device<->head transforms
mne.io.concatenate_raws(
    [raw, read_raw_ctf(raw_fname2)], on_mismatch='ignore')
raw_erm = read_raw_ctf(erm_fname)


I am working on the file "0.24/_downloads/fcc5782db3e2930fc79f31bc745495ed/60_ctf_bst_auditory.ipynb".
The first task is:
Show activations as topography figures.
Can you write Python code for it?

times = np.arange(0.05, 0.301, 0.025)
evoked_std.plot_topomap(times=times, title='Standard', time_unit='s')

evoked_dev.plot_topomap(times=times, title='Deviant', time_unit='s')


I am working on the file "0.24/_downloads/fcc5782db3e2930fc79f31bc745495ed/60_ctf_bst_auditory.ipynb".
The first task is:
The data was collected with a CTF 275 system at 2400 Hz and low-pass
filtered at 600 Hz. Here the data and empty room data files are read to
construct instances of :class:mne.io.Raw.
Can you write Python code for it?

data_path = bst_auditory.data_path()

subject = 'bst_auditory'
subjects_dir = op.join(data_path, 'subjects')

raw_fname1 = op.join(data_path, 'MEG', subject, 'S01_AEF_20131218_01.ds')
raw_fname2 = op.join(data_path, 'MEG', subject, 'S01_AEF_20131218_02.ds')
erm_fname = op.join(data_path, 'MEG', subject, 'S01_Noise_20131218_01.ds')


I am working on the file "0.24/_downloads/fcc5782db3e2930fc79f31bc745495ed/60_ctf_bst_auditory.ipynb".
The first task is:
The epochs (trials) are created for MEG channels. First we find the picks
for MEG and EOG channels. Then the epochs are constructed using these picks.
The epochs overlapping with annotated bad segments are also rejected by
default. To turn off rejection by bad segments (as was done earlier with
saccades) you can use keyword reject_by_annotation=False.
Can you write Python code for it?

epochs = mne.Epochs(raw, events, event_id, tmin, tmax, picks=['meg', 'eog'],
                    baseline=(None, 0), reject=reject, preload=False,
                    proj=True)


I am working on the file "0.24/_downloads/fcc5782db3e2930fc79f31bc745495ed/60_ctf_bst_auditory.ipynb".
The first task is:
The event timing is adjusted by comparing the trigger times on detected
sound onsets on channel UADC001-4408.
Can you write Python code for it?

sound_data = raw[raw.ch_names.index('UADC001-4408')][0][0]
onsets = np.where(np.abs(sound_data) > 2. * np.std(sound_data))[0]
min_diff = int(0.5 * raw.info['sfreq'])
diffs = np.concatenate([[min_diff + 1], np.diff(onsets)])
onsets = onsets[diffs > min_diff]
assert len(onsets) == len(events)
diffs = 1000. * (events[:, 0] - onsets) / raw.info['sfreq']
print('Trigger delay removed (μ ± σ): %0.1f ± %0.1f ms'
      % (np.mean(diffs), np.std(diffs)))
events[:, 0] = onsets
del sound_data, diffs


I am working on the file "0.24/_downloads/fcc5782db3e2930fc79f31bc745495ed/60_ctf_bst_auditory.ipynb".
The first task is:
The sources are computed using dSPM method and plotted on an inflated brain
surface. For interactive controls over the image, use keyword
time_viewer=True.
Standard condition.
Can you write Python code for it?

stc_standard = mne.minimum_norm.apply_inverse(evoked_std, inv, lambda2, 'dSPM')
brain = stc_standard.plot(subjects_dir=subjects_dir, subject=subject,
                          surface='inflated', time_viewer=False, hemi='lh',
                          initial_time=0.1, time_unit='s')
del stc_standard, brain


I am working on the file "0.24/_downloads/fcc5782db3e2930fc79f31bc745495ed/60_ctf_bst_auditory.ipynb".
The first task is:
To reduce memory consumption and running time, some of the steps are
precomputed. To run everything from scratch change use_precomputed to
False. With use_precomputed = False running time of this script can
be several minutes even on a fast computer.
Can you write Python code for it?


I am working on the file "0.24/_downloads/fcc5782db3e2930fc79f31bc745495ed/60_ctf_bst_auditory.ipynb".
The first task is:
Typical preprocessing step is the removal of power line artifact (50 Hz or
60 Hz). Here we lowpass filter the data at 40 Hz, which will remove all
line artifacts (and high frequency information). Normally this would be done
to raw data (with :func:mne.io.Raw.filter), but to reduce memory
consumption of this tutorial, we do it at evoked stage. (At the raw stage,
you could alternatively notch filter with :func:mne.io.Raw.notch_filter.)
Can you write Python code for it?

for evoked in (evoked_std, evoked_dev):
    evoked.filter(l_freq=None, h_freq=40., fir_design='firwin')


I am working on the file "0.24/_downloads/fcc5782db3e2930fc79f31bc745495ed/60_ctf_bst_auditory.ipynb".
The first task is:
Typical preprocessing step is the removal of power line artifact (50 Hz or
60 Hz). Here we notch filter the data at 60, 120 and 180 to remove the
original 60 Hz artifact and the harmonics. The power spectra are plotted
before and after the filtering to show the effect. The drop after 600 Hz
appears because the data was filtered during the acquisition. In memory
saving mode we do the filtering at evoked stage, which is not something you
usually would do.
Can you write Python code for it?

if not use_precomputed:
    raw.plot_psd(tmax=np.inf, picks='meg')
    notches = np.arange(60, 181, 60)
    raw.notch_filter(notches, phase='zero-double', fir_design='firwin2')
    raw.plot_psd(tmax=np.inf, picks='meg')


I am working on the file "0.24/_downloads/fcc5782db3e2930fc79f31bc745495ed/60_ctf_bst_auditory.ipynb".
The first task is:
Visually inspect the effects of projections. Click on 'proj' button at the
bottom right corner to toggle the projectors on/off. EOG events can be
plotted by adding the event list as a keyword argument. As the bad segments
and saccades were added as annotations to the raw data, they are plotted as
well.
Can you write Python code for it?


I am working on the file "0.24/_downloads/fcc5782db3e2930fc79f31bc745495ed/60_ctf_bst_auditory.ipynb".
The first task is:
We can see the MMN effect more clearly by looking at the difference between
the two conditions. P50 and N100 are no longer visible, but MMN/P200 and
P300 are emphasised.
Can you write Python code for it?

evoked_difference = combine_evoked([evoked_dev, evoked_std], weights=[1, -1])
evoked_difference.plot(window_title='Difference', gfp=True, time_unit='s')


I am working on the file "0.24/_downloads/fcc5782db3e2930fc79f31bc745495ed/60_ctf_bst_auditory.ipynb".
The first task is:
We mark a set of bad channels that seem noisier than others. This can also
be done interactively with raw.plot by clicking the channel name
(or the line). The marked channels are added as bad when the browser window
is closed.
Can you write Python code for it?


I am working on the file "0.24/_downloads/fcc5782db3e2930fc79f31bc745495ed/60_ctf_bst_auditory.ipynb".
The first task is:
We only use first 40 good epochs from each run. Since we first drop the bad
epochs, the indices of the epochs are no longer same as in the original
epochs collection. Investigation of the event timings reveals that first
epoch from the second run corresponds to index 182.
Can you write Python code for it?

epochs.drop_bad()
epochs_standard = mne.concatenate_epochs([epochs['standard'][range(40)],
                                          epochs['standard'][182:222]])
epochs_standard.load_data()  # Resampling to save memory.
epochs_standard.resample(600, npad='auto')
epochs_deviant = epochs['deviant'].load_data()
epochs_deviant.resample(600, npad='auto')
del epochs


I am working on the file "0.24/_downloads/fcc5782db3e2930fc79f31bc745495ed/60_ctf_bst_auditory.ipynb".
The first task is:
Working with CTF data: the Brainstorm auditory dataset
Here we compute the evoked from raw for the auditory Brainstorm
tutorial dataset. For comparison, see :footcite:TadelEtAl2011 and the
associated brainstorm site
<https://neuroimage.usc.edu/brainstorm/Tutorials/Auditory>_.
Experiment:
- One subject, 2 acquisition runs 6 minutes each.
- Each run contains 200 regular beeps and 40 easy deviant beeps.
- Random ISI: between 0.7s and 1.7s seconds, uniformly distributed.
- Button pressed when detecting a deviant with the right index finger.

The specifications of this dataset were discussed initially on the
FieldTrip bug tracker
<http://bugzilla.fieldtriptoolbox.org/show_bug.cgi?id=2300>__.
Can you write Python code for it?

# Authors: Mainak Jas <mainak.jas@telecom-paristech.fr>
#          Eric Larson <larson.eric.d@gmail.com>
#          Jaakko Leppakangas <jaeilepp@student.jyu.fi>
#
# License: BSD-3-Clause

import os.path as op
import pandas as pd
import numpy as np

import mne
from mne import combine_evoked
from mne.minimum_norm import apply_inverse
from mne.datasets.brainstorm import bst_auditory
from mne.io import read_raw_ctf

print(__doc__)


I am working on the file "0.7/notebooks/auto_examples/ask-and-tell.ipynb".
The first task is:
By using the :class:Optimizer class directly you get control over the
optimization loop.
You can also pickle your :class:Optimizer instance if you want to end the
process running it and resume it later. This is handy if your experiment
takes a very long time and you want to shutdown your computer in the
meantime:
Can you write Python code for it?

import pickle

with open('my-optimizer.pkl', 'wb') as f:
    pickle.dump(opt, f)

with open('my-optimizer.pkl', 'rb') as f:
    opt_restored = pickle.load(f)


I am working on the file "0.7/notebooks/auto_examples/ask-and-tell.ipynb".
The first task is:
Here a quick plot to visualize what the function looks like:
Can you write Python code for it?

# Plot f(x) + contours
plt.set_cmap("viridis")
x = np.linspace(-2, 2, 400).reshape(-1, 1)
fx = np.array([objective(x_i, noise_level=0.0) for x_i in x])
plt.plot(x, fx, "r--", label="True (unknown)")
plt.fill(np.concatenate([x, x[::-1]]),
         np.concatenate(([fx_i - 1.9600 * noise_level for fx_i in fx],
                         [fx_i + 1.9600 * noise_level for fx_i in fx[::-1]])),
         alpha=.2, fc="r", ec="None")
plt.legend()
plt.grid()
plt.show()


I am working on the file "0.7/notebooks/auto_examples/ask-and-tell.ipynb".
The first task is:
Let us sample a few more points and plot the optimizer again:
Can you write Python code for it?

for i in range(10):
    next_x = opt.ask()
    f_val = objective(next_x)
    opt.tell(next_x, f_val)

plot_optimizer(opt, x, fx)


I am working on the file "0.7/notebooks/auto_examples/ask-and-tell.ipynb".
The first task is:
Like *_minimize() the first few points are random suggestions as there
is no data yet with which to fit a surrogate model.
Can you write Python code for it?

for i in range(9):
    next_x = opt.ask()
    f_val = objective(next_x)
    opt.tell(next_x, f_val)


I am working on the file "0.7/notebooks/auto_examples/ask-and-tell.ipynb".
The first task is:
Our 1D toy problem, this is the function we are trying to
minimize
Can you write Python code for it?

def objective(x, noise_level=noise_level):
    return np.sin(5 * x[0]) * (1 - np.tanh(x[0] ** 2))\
           + np.random.randn() * noise_level


I am working on the file "0.7/notebooks/auto_examples/ask-and-tell.ipynb".
The first task is:
The Setup
We will use a simple 1D problem to illustrate the API. This is a little bit
artificial as you normally would not use the ask-and-tell interface if you
had a function you can call to evaluate the objective.
Can you write Python code for it?

from skopt.learning import ExtraTreesRegressor
from skopt import Optimizer

noise_level = 0.1


I am working on the file "0.7/notebooks/auto_examples/ask-and-tell.ipynb".
The first task is:
We can now plot the random suggestions and the first model that has been
fit:
Can you write Python code for it?

from skopt.acquisition import gaussian_ei

def plot_optimizer(opt, x, fx):
    model = opt.models[-1]
    x_model = opt.space.transform(x.tolist())

    # Plot true function.
    plt.plot(x, fx, "r--", label="True (unknown)")
    plt.fill(np.concatenate([x, x[::-1]]),
             np.concatenate([fx - 1.9600 * noise_level,
                             fx[::-1] + 1.9600 * noise_level]),
             alpha=.2, fc="r", ec="None")

    # Plot Model(x) + contours
    y_pred, sigma = model.predict(x_model, return_std=True)
    plt.plot(x, y_pred, "g--", label=r"$\mu(x)$")
    plt.fill(np.concatenate([x, x[::-1]]),
             np.concatenate([y_pred - 1.9600 * sigma,
                             (y_pred + 1.9600 * sigma)[::-1]]),
             alpha=.2, fc="g", ec="None")

    # Plot sampled points
    plt.plot(opt.Xi, opt.yi,
             "r.", markersize=8, label="Observations")

    acq = gaussian_ei(x_model, model, y_opt=np.min(opt.yi))
    # shift down to make a better plot
    acq = 4 * acq - 2
    plt.plot(x, acq, "b", label="EI(x)")
    plt.fill_between(x.ravel(), -2.0, acq.ravel(), alpha=0.3, color='blue')

    # Adjust plot layout
    plt.grid()
    plt.legend(loc='best')

plot_optimizer(opt, x, fx)


I am working on the file "0.7/notebooks/auto_examples/bayesian-optimization.ipynb".
The first task is:
For further inspection of the results, attributes of the res named tuple
provide the following information:

x [float]: location of the minimum.
fun [float]: function value at the minimum.
models: surrogate models used for each iteration.
x_iters [array]:
   location of function evaluation for each iteration.
func_vals [array]: function value for each iteration.
space [Space]: the optimization space.
specs [dict]: parameters passed to the function.
Can you write Python code for it?


I am working on the file "0.7/notebooks/auto_examples/bayesian-optimization.ipynb".
The first task is:
Let us now visually examine

The approximation of the fit gp model to the original function.
The acquisition values that determine the next point to be queried.
Can you write Python code for it?

from skopt.acquisition import gaussian_ei

plt.rcParams["figure.figsize"] = (8, 14)

x = np.linspace(-2, 2, 400).reshape(-1, 1)
x_gp = res.space.transform(x.tolist())
fx = np.array([f(x_i, noise_level=0.0) for x_i in x])


I am working on the file "0.7/notebooks/auto_examples/bayesian-optimization.ipynb".
The first task is:
Note. In skopt, functions $f$ are assumed to take as input a 1D
vector $x$: represented as an array-like and to return a scalar
$f(x)$:.
Can you write Python code for it?

# Plot f(x) + contours
x = np.linspace(-2, 2, 400).reshape(-1, 1)
fx = [f(x_i, noise_level=0.0) for x_i in x]
plt.plot(x, fx, "r--", label="True (unknown)")
plt.fill(np.concatenate([x, x[::-1]]),
         np.concatenate(([fx_i - 1.9600 * noise_level for fx_i in fx],
                         [fx_i + 1.9600 * noise_level for fx_i in fx[::-1]])),
         alpha=.2, fc="r", ec="None")
plt.legend()
plt.grid()
plt.show()


I am working on the file "0.7/notebooks/auto_examples/bayesian-optimization.ipynb".
The first task is:
Plot the 5 iterations following the 5 random points
Can you write Python code for it?

for n_iter in range(5):
    gp = res.models[n_iter]
    curr_x_iters = res.x_iters[:5+n_iter]
    curr_func_vals = res.func_vals[:5+n_iter]

    # Plot true function.
    plt.subplot(5, 2, 2*n_iter+1)
    plt.plot(x, fx, "r--", label="True (unknown)")
    plt.fill(np.concatenate([x, x[::-1]]),
             np.concatenate([fx - 1.9600 * noise_level,
                             fx[::-1] + 1.9600 * noise_level]),
             alpha=.2, fc="r", ec="None")

    # Plot GP(x) + contours
    y_pred, sigma = gp.predict(x_gp, return_std=True)
    plt.plot(x, y_pred, "g--", label=r"$\mu_{GP}(x)$")
    plt.fill(np.concatenate([x, x[::-1]]),
             np.concatenate([y_pred - 1.9600 * sigma,
                             (y_pred + 1.9600 * sigma)[::-1]]),
             alpha=.2, fc="g", ec="None")

    # Plot sampled points
    plt.plot(curr_x_iters, curr_func_vals,
             "r.", markersize=8, label="Observations")

    # Adjust plot layout
    plt.grid()

    if n_iter == 0:
        plt.legend(loc="best", prop={'size': 6}, numpoints=1)

    if n_iter != 4:
        plt.tick_params(axis='x', which='both', bottom='off',
                        top='off', labelbottom='off')

    # Plot EI(x)
    plt.subplot(5, 2, 2*n_iter+2)
    acq = gaussian_ei(x_gp, gp, y_opt=np.min(curr_func_vals))
    plt.plot(x, acq, "b", label="EI(x)")
    plt.fill_between(x.ravel(), -2.0, acq.ravel(), alpha=0.3, color='blue')

    next_x = res.x_iters[5+n_iter]
    next_acq = gaussian_ei(res.space.transform([next_x]), gp,
                           y_opt=np.min(curr_func_vals))
    plt.plot(next_x, next_acq, "bo", markersize=6, label="Next query point")

    # Adjust plot layout
    plt.ylim(0, 0.1)
    plt.grid()

    if n_iter == 0:
        plt.legend(loc="best", prop={'size': 6}, numpoints=1)

    if n_iter != 4:
        plt.tick_params(axis='x', which='both', bottom='off',
                        top='off', labelbottom='off')

plt.show()


I am working on the file "0.7/notebooks/auto_examples/bayesian-optimization.ipynb".
The first task is:
Together these attributes can be used to visually inspect the results of the
minimization, such as the convergence trace or the acquisition function at
the last iteration:
Can you write Python code for it?

from skopt.plots import plot_convergence
plot_convergence(res);


I am working on the file "0.7/notebooks/auto_examples/exploration-vs-exploitation.ipynb".
The first task is:
And for xi when using acq_func="EI": or acq_func="PI":
Can you write Python code for it?

opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=1,
                acq_func="PI", acq_optimizer="sampling",
                acq_func_kwargs=acq_func_kwargs)

opt.run(objective, n_iter=20)
plot_optimizer(opt, x, fx)


I am working on the file "0.7/notebooks/auto_examples/exploration-vs-exploitation.ipynb".
The first task is:
Changing kappa and xi on the go
If we want to change kappa or ki at any point during our optimization
process we just replace opt.acq_func_kwargs. Remember to call
opt.update_next() after the change, in order for next point to be
recalculated.
Can you write Python code for it?

acq_func_kwargs = {"kappa": 0}

opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=1,
                acq_func="LCB", acq_optimizer="sampling",
                acq_func_kwargs=acq_func_kwargs)

opt.acq_func_kwargs

opt.run(objective, n_iter=20)
plot_optimizer(opt, x, fx)

acq_func_kwargs = {"kappa": 100000}

opt.acq_func_kwargs = acq_func_kwargs
opt.update_next()

opt.run(objective, n_iter=20)
plot_optimizer(opt, x, fx)


I am working on the file "0.7/notebooks/auto_examples/exploration-vs-exploitation.ipynb".
The first task is:
Exploration vs exploitation
Sigurd Carlen, September 2019.
Reformatted by Holger Nahrstaedt 2020
.. currentmodule:: skopt
We can control how much the acqusition function favors exploration and
exploitation by tweaking the two parameters kappa and xi. Higher values
means more exploration and less exploitation and vice versa with low values.
kappa is only used if acq_func is set to "LCB". xi is used when acq_func is
"EI" or "PI". By default the acqusition function is set to "gp_hedge" which
chooses the best of these three. Therefore I recommend not using gp_hedge
when tweaking exploration/exploitation, but instead choosing "LCB",
"EI" or "PI.
The way to pass kappa and xi to the optimizer is to use the named argument
"acq_func_kwargs". This is a dict of extra arguments for the aqcuisittion
function.
If you want opt.ask() to give a new acquisition value imdediatly after
tweaking kappa or xi call opt.update_next(). This ensures that the next
value is updated with the new acquisition parameters.
Can you write Python code for it?

print(__doc__)

import numpy as np
np.random.seed(1234)
import matplotlib.pyplot as plt


I am working on the file "0.7/notebooks/auto_examples/exploration-vs-exploitation.ipynb".
The first task is:
Note that negative values does not work with the "PI"-acquisition function
but works with "EI":
Can you write Python code for it?

acq_func_kwargs = {"xi": -1000000000000}

opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=1,
                acq_func="PI", acq_optimizer="sampling",
                acq_func_kwargs=acq_func_kwargs)

opt.run(objective, n_iter=20)
plot_optimizer(opt, x, fx)

opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=1,
                acq_func="EI", acq_optimizer="sampling",
                acq_func_kwargs=acq_func_kwargs)

opt.run(objective, n_iter=20)
plot_optimizer(opt, x, fx)


I am working on the file "0.7/notebooks/auto_examples/exploration-vs-exploitation.ipynb".
The first task is:
We can also favor exploitaton:
Can you write Python code for it?

acq_func_kwargs = {"xi": 0.000001, "kappa": 0.001}

opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=1,
                acq_func="LCB", acq_optimizer="sampling",
                acq_func_kwargs=acq_func_kwargs)

opt.run(objective, n_iter=20)
plot_optimizer(opt, x, fx)

opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=1,
                acq_func="EI", acq_optimizer="sampling",
                acq_func_kwargs=acq_func_kwargs)

opt.run(objective, n_iter=20)
plot_optimizer(opt, x, fx)

opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=1,
                acq_func="PI", acq_optimizer="sampling",
                acq_func_kwargs=acq_func_kwargs)

opt.run(objective, n_iter=20)
plot_optimizer(opt, x, fx)


I am working on the file "0.7/notebooks/auto_examples/exploration-vs-exploitation.ipynb".
The first task is:
We run a an optimization loop with standard settings
Can you write Python code for it?

for i in range(30):
    next_x = opt.ask()
    f_val = objective(next_x)
    opt.tell(next_x, f_val)
# The same output could be created with opt.run(objective, n_iter=30)
plot_optimizer(opt, x, fx)


I am working on the file "0.7/notebooks/auto_examples/exploration-vs-exploitation.ipynb".
The first task is:
We see that some minima is found and "exploited"
Now lets try to set kappa and xi using'to other values and
pass it to the optimizer:
Can you write Python code for it?

acq_func_kwargs = {"xi": 10000, "kappa": 10000}

opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=1,
                acq_optimizer="sampling",
                acq_func_kwargs=acq_func_kwargs)

opt.run(objective, n_iter=20)
plot_optimizer(opt, x, fx)


I am working on the file "0.7/notebooks/auto_examples/exploration-vs-exploitation.ipynb".
The first task is:
We see that the points are more random now.
This works both for kappa when using acq_func="LCB":
Can you write Python code for it?

opt = Optimizer([(-2.0, 2.0)], "GP", n_initial_points=1,
                acq_func="LCB", acq_optimizer="sampling",
                acq_func_kwargs=acq_func_kwargs)

opt.run(objective, n_iter=20)
plot_optimizer(opt, x, fx)


I am working on the file "0.7/notebooks/auto_examples/hyperparameter-optimization.ipynb".
The first task is:
Convergence plot
Can you write Python code for it?

from skopt.plots import plot_convergence

plot_convergence(res_gp)


I am working on the file "0.7/notebooks/auto_examples/hyperparameter-optimization.ipynb".
The first task is:
Next, we need to define the bounds of the dimensions of the search space
we want to explore and pick the objective. In this case the cross-validation
mean absolute error of a gradient boosting regressor over the Boston
dataset, as a function of its hyper-parameters.
Can you write Python code for it?

from skopt.space import Real, Integer
from skopt.utils import use_named_args

# The list of hyper-parameters we want to optimize. For each one we define the
# bounds, the corresponding scikit-learn parameter name, as well as how to
# sample values from that dimension (`'log-uniform'` for the learning rate)
space  = [Integer(1, 5, name='max_depth'),
          Real(10**-5, 10**0, "log-uniform", name='learning_rate'),
          Integer(1, n_features, name='max_features'),
          Integer(2, 100, name='min_samples_split'),
          Integer(1, 100, name='min_samples_leaf')]

# this decorator allows your objective function to receive a the parameters as
# keyword arguments. This is particularly convenient when you want to set
# scikit-learn estimator parameters
@use_named_args(space)
def objective(**params):
    reg.set_params(**params)

    return -np.mean(cross_val_score(reg, X, y, cv=5, n_jobs=-1,
                                    scoring="neg_mean_absolute_error"))


I am working on the file "0.7/notebooks/auto_examples/hyperparameter-optimization.ipynb".
The first task is:
Objective
To tune the hyper-parameters of our model we need to define a model,
decide which parameters to optimize, and define the objective function
we want to minimize.
Can you write Python code for it?

from sklearn.datasets import load_boston
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import cross_val_score

boston = load_boston()
X, y = boston.data, boston.target
n_features = X.shape[1]

# gradient boosted trees tend to do well on problems like this
reg = GradientBoostingRegressor(n_estimators=50, random_state=0)


I am working on the file "0.7/notebooks/auto_examples/interruptible-optimization.ipynb".
The first task is:
Continue the search
The previous results can then be used to continue the optimization process:
Can you write Python code for it?

x0 = res.x_iters
y0 = res.func_vals

gp_minimize(obj_fun,            # the function to minimize
              [(-20.0, 20.0)],    # the bounds on each dimension of x
              x0=x0,              # already examined values for x
              y0=y0,              # observed values for x0
              acq_func="LCB",     # the acquisition function (optional)
              n_calls=10,         # the number of evaluations of f including at x0
              n_random_starts=0,  # the number of random initialization points
              callback=[checkpoint_saver],
              random_state=777);


I am working on the file "0.7/notebooks/auto_examples/interruptible-optimization.ipynb".
The first task is:
Interruptible optimization runs with checkpoints
Christian Schell, Mai 2018
Reformatted by Holger Nahrstaedt 2020
.. currentmodule:: skopt
Problem statement
Optimization runs can take a very long time and even run for multiple days.
If for some reason the process has to be interrupted results are irreversibly
lost, and the routine has to start over from the beginning.
With the help of the :class:callbacks.CheckpointSaver callback the optimizer's current state
can be saved after each iteration, allowing to restart from that point at any
time.
This is useful, for example,

if you don't know how long the process will take and cannot hog computational resources forever
if there might be system failures due to shaky infrastructure (or colleagues...)
if you want to adjust some parameters and continue with the already obtained results
Can you write Python code for it?

print(__doc__)
import sys
import numpy as np
np.random.seed(777)
import os

# The followings are hacks to allow sphinx-gallery to run the example.
sys.path.insert(0, os.getcwd())
main_dir = os.path.basename(sys.modules['__main__'].__file__)
IS_RUN_WITH_SPHINX_GALLERY = main_dir != os.getcwd()


I am working on the file "0.7/notebooks/auto_examples/interruptible-optimization.ipynb".
The first task is:
Simple example
We will use pretty much the same optimization problem as in the
sphx_glr_auto_examples_bayesian-optimization.py
notebook. Additionally we will instantiate the :class:callbacks.CheckpointSaver
and pass it to the minimizer:
Can you write Python code for it?

from skopt import gp_minimize
from skopt import callbacks
from skopt.callbacks import CheckpointSaver

noise_level = 0.1

if IS_RUN_WITH_SPHINX_GALLERY:
    # When this example is run with sphinx gallery, it breaks the pickling
    # capacity for multiprocessing backend so we have to modify the way we
    # define our functions. This has nothing to do with the example.
    from utils import obj_fun
else:
    def obj_fun(x, noise_level=noise_level):
        return np.sin(5 * x[0]) * (1 - np.tanh(x[0] ** 2)) + np.random.randn() * noise_level

checkpoint_saver = CheckpointSaver("./checkpoint.pkl", compress=9) # keyword arguments will be passed to `skopt.dump`

gp_minimize(obj_fun,                       # the function to minimize
              [(-20.0, 20.0)],             # the bounds on each dimension of x
              x0=[-20.],                     # the starting point
              acq_func="LCB",              # the acquisition function (optional)
              n_calls=10,                   # the number of evaluations of f including at x0
              n_random_starts=0,           # the number of random initialization points
              callback=[checkpoint_saver], # a list of callbacks including the checkpoint saver
              random_state=777);


I am working on the file "0.7/notebooks/auto_examples/optimizer-with-different-base-estimator.ipynb".
The first task is:
GP kernel
Can you write Python code for it?

fig = plt.figure()
fig.suptitle("Standard GP kernel")
for i in range(10):
    next_x = opt_gp.ask()
    f_val = objective(next_x)
    res = opt_gp.tell(next_x, f_val)
    if i >= 5:
        plot_optimizer(res, opt_gp._next_x, x, fx, n_iter=i-5, max_iters=5)
plt.tight_layout(rect=[0, 0.03, 1, 0.95])
plt.plot()


I am working on the file "0.7/notebooks/auto_examples/optimizer-with-different-base-estimator.ipynb".
The first task is:
Toy example
Let assume the following noisy function $f$:
Can you write Python code for it?

noise_level = 0.1

# Our 1D toy problem, this is the function we are trying to
# minimize

def objective(x, noise_level=noise_level):
    return np.sin(5 * x[0]) * (1 - np.tanh(x[0] ** 2))\
           + np.random.randn() * noise_level

from skopt import Optimizer
opt_gp = Optimizer([(-2.0, 2.0)], base_estimator="GP", n_initial_points=5,
                acq_optimizer="sampling", random_state=42)

x = np.linspace(-2, 2, 400).reshape(-1, 1)
fx = np.array([objective(x_i, noise_level=0.0) for x_i in x])

from skopt.acquisition import gaussian_ei

def plot_optimizer(res, next_x, x, fx, n_iter, max_iters=5):
    x_gp = res.space.transform(x.tolist())
    gp = res.models[-1]
    curr_x_iters = res.x_iters
    curr_func_vals = res.func_vals

    # Plot true function.
    ax = plt.subplot(max_iters, 2, 2 * n_iter + 1)
    plt.plot(x, fx, "r--", label="True (unknown)")
    plt.fill(np.concatenate([x, x[::-1]]),
             np.concatenate([fx - 1.9600 * noise_level,
                             fx[::-1] + 1.9600 * noise_level]),
             alpha=.2, fc="r", ec="None")
    if n_iter < max_iters - 1:
        ax.get_xaxis().set_ticklabels([])
    # Plot GP(x) + contours
    y_pred, sigma = gp.predict(x_gp, return_std=True)
    plt.plot(x, y_pred, "g--", label=r"$\mu_{GP}(x)$")
    plt.fill(np.concatenate([x, x[::-1]]),
             np.concatenate([y_pred - 1.9600 * sigma,
                             (y_pred + 1.9600 * sigma)[::-1]]),
             alpha=.2, fc="g", ec="None")

    # Plot sampled points
    plt.plot(curr_x_iters, curr_func_vals,
             "r.", markersize=8, label="Observations")
    plt.title(r"x* = %.4f, f(x*) = %.4f" % (res.x[0], res.fun))
    # Adjust plot layout
    plt.grid()

    if n_iter == 0:
        plt.legend(loc="best", prop={'size': 6}, numpoints=1)

    if n_iter != 4:
        plt.tick_params(axis='x', which='both', bottom='off',
                        top='off', labelbottom='off')

    # Plot EI(x)
    ax = plt.subplot(max_iters, 2, 2 * n_iter + 2)
    acq = gaussian_ei(x_gp, gp,...


I am working on the file "0.7/notebooks/auto_examples/optimizer-with-different-base-estimator.ipynb".
The first task is:
Use different base estimators for optimization
Sigurd Carlen, September 2019.
Reformatted by Holger Nahrstaedt 2020
.. currentmodule:: skopt
To use different base_estimator or create a regressor with different parameters,
we can create a regressor object and set it as kernel.
Can you write Python code for it?


I am working on the file "0.7/notebooks/auto_examples/parallel-optimization.ipynb".
The first task is:
Parallel optimization
Iaroslav Shcherbatyi, May 2017.
Reviewed by Manoj Kumar and Tim Head.
Reformatted by Holger Nahrstaedt 2020
.. currentmodule:: skopt
Introduction
For many practical black box optimization problems expensive objective can be
evaluated in parallel at multiple points. This allows to get more objective
evaluations per unit of time, which reduces the time necessary to reach good
objective values when appropriate optimization algorithms are used, see for
example results in [1]_ and the references therein.
One such example task is a selection of number and activation function of a
neural network which results in highest accuracy for some machine learning
problem. For such task, multiple neural networks with different combinations
of number of neurons and activation function type can be evaluated at the same
time in parallel on different cpu cores / computational nodes.
The “ask and tell” API of scikit-optimize exposes functionality that allows to
obtain multiple points for evaluation in parallel. Intended usage of this
interface is as follows:

Initialize instance of the Optimizer class from skopt
Obtain n points for evaluation in parallel by calling the ask method of an optimizer instance with the n_points argument set to n > 0
Evaluate points
Provide points and corresponding objectives using the tell method of an optimizer instance
Continue from step 2 until eg maximum number of evaluations reached
Can you write Python code for it?


I am working on the file "0.7/notebooks/auto_examples/plots/partial-dependence-plot-with-categorical.ipynb".
The first task is:
Bayesian optimization
Can you write Python code for it?

SPACE = [
    Integer(1, 20, name='max_depth'),
    Integer(2, 100, name='min_samples_split'),
    Integer(5, 30, name='min_samples_leaf'),
    Integer(1, 30, name='max_features'),
    Categorical(list('abc'), name='dummy'),
    Categorical(['gini', 'entropy'], name='criterion'),
    Categorical(list('def'), name='dummy'),
]

result = gp_minimize(objective, SPACE, n_calls=20)


I am working on the file "0.7/notebooks/auto_examples/plots/partial-dependence-plot-with-categorical.ipynb".
The first task is:
Modify the shown minimum
Here we try with setting the other parameters to something other than
"result". When dealing with categorical dimensions we can't use
'expected_minimum'. Therefore we try with "expected_minimum_random"
which is a naive way of finding the minimum of the surrogate by only
using random sampling. n_minimum_search sets the number of random samples,
which is used to find the minimum
Can you write Python code for it?

_ = plot_objective(result, n_points=10, sample_source='expected_minimum_random',
                   minimum='expected_minimum_random', n_minimum_search=10000)


I am working on the file "0.7/notebooks/auto_examples/plots/partial-dependence-plot-with-categorical.ipynb".
The first task is:
Partial Dependence Plots  with categorical values
Sigurd Carlsen Feb 2019
Holger Nahrstaedt 2020
.. currentmodule:: skopt
Plot objective now supports optional use of partial dependence as well as
different methods of defining parameter values for dependency plots.
Can you write Python code for it?

print(__doc__)
import sys
from skopt.plots import plot_objective
from skopt import forest_minimize
import numpy as np
np.random.seed(123)
import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import cross_val_score
from skopt.space import Integer, Categorical
from skopt import plots, gp_minimize
from skopt.plots import plot_objective


I am working on the file "0.7/notebooks/auto_examples/plots/partial-dependence-plot-with-categorical.ipynb".
The first task is:
Partial dependence plot
Here we see an example of using partial dependence. Even when setting
n_points all the way down to 10 from the default of 40, this method is
still very slow. This is because partial dependence calculates 250 extra
predictions for each point on the plots.
Can you write Python code for it?


I am working on the file "0.7/notebooks/auto_examples/plots/partial-dependence-plot-with-categorical.ipynb".
The first task is:
Set a minimum location
Lastly we can also define these parameters ourselfs by
parsing a list as the pars argument:
Can you write Python code for it?

_ = plot_objective(result, n_points=10, sample_source=[15, 4, 7, 15, 'b', 'entropy', 'e'],
                   minimum=[15, 4, 7, 15, 'b', 'entropy', 'e'])


I am working on the file "0.7/notebooks/auto_examples/plots/partial-dependence-plot.ipynb".
The first task is:
Modify the shown minimum
Here we try with setting the minimum parameters to something other than
"result". First we try with "expected_minimum" which is the set of
parameters that gives the miniumum value of the surrogate function,
using scipys minimum search method.
Can you write Python code for it?

_ = plot_objective(result, n_points=10, sample_source='expected_minimum',
                   minimum='expected_minimum')


I am working on the file "0.7/notebooks/auto_examples/plots/partial-dependence-plot.ipynb".
The first task is:
Objective function
Plot objective now supports optional use of partial dependence as well as
different methods of defining parameter values for dependency plots
Can you write Python code for it?

# Here we define a function that we evaluate.
def funny_func(x):
    s = 0
    for i in range(len(x)):
        s += (x[i] * i) ** 2
    return s