I am working on the file "Sessions/Session14/Day3/AutoencodersBlank.ipynb".
The first task is:
Problem 4a. Understand our Results
Plot an image (remember you will need to reshape it to a 14x14 grid) with imshow, and plot the autoencoder output for the same galaxy. Try plotting the difference between the two. What does your algorithm do well reconstructing? Are there certain features which it fails to reproduce?
Can you write Python code for it?

#Make an image of the original image

#Make an image of its reconstruction

#Make an image of (original - reconstruction)


I am working on the file "Sessions/Session14/Day3/AutoencodersBlank.ipynb".
The first task is:
Problem 4b.
Make a scatter plot of two of the 10 latent space dimensions. Do you notice any interesting correlations between different subsets of the latent space? Any interesting clustering?
Try color coding each point by the galaxy label using plt.scatter
Can you write Python code for it?

#Scatter plot between two dimensions of the latent space
#Try coloring the points


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders.ipynb".
The first task is:
A few things you might have noticed and that can be useful to keep in mind:

The model expects a single precision input. You can change the type of a tensor with tensor_name.type(), where tensor_name is the name of your tensor and type is the dtype. For typecasting into single precision floating points, use float(). A numpy array is typecasted with array_name.astype(type). For single precision, the type should be np.float32.
Before we analyze tensors we often want to convert them to numpy arrays with tensor_name.numpy()
If pytorch has been tracking operations that resulted in the current tensor value, you need to detach the tensor from the graph before you can transform it into a numpy array: tensor_name.detach(). Scalars can be detached with scalar.item()
If you tensor is currently on the GPU, you can bring it onto the CPU with tensor_name.cpu()

Variational Autoencoder
Your task today is to transform the above Autoencoder into a Variational Autoencoder. You can follow either follow my Step-by-Step instructions or do it your way (there's more than one solution).
We start by modifying the encoder network. Tasks are marked with the keyword #TASK, hints are marked with the keyword #HINT. Question marks are used to indicate places where you need to fill in code in order to recover the solutions. You don't have to follow the structure of the solutions, there's more than one way to code a VAE. You can also replace the statements with your own code if you find your approach more intuitive!
STEP 1: Modify the encoder network.
The encoder network is used to characterize the variational distribution q(z|x). Recall that we want q(z|x) to be a Gaussian with diagonal covariance. An N-dimensional Gaussian with diagonal covariance is eqivalent to N independent 1-dimensional Gaussians (where N is the latent size). Each Gaussian is defined by two quantities, its mean and variance (or, if we take the sqrt, the standard deviation).
Can you write Python code for it?

class VAEEncoder(nn.Module):

    def __init__(self, seed=853):
        super(VAEEncoder, self).__init__()
        #TASK: change the output size of the encoder network. How many parameters must it return to define q(z|x)?
        self.seed = torch.manual_seed(seed)
        self.fc1 = nn.Linear(INPUT_SIZE,50)
        self.fc2 = nn.Linear(50,??)

    def forward(self, x):
        #TASK: change the output of the encoder network. Instead of just returning z, it should return z and ...?
        #HINT: Don't forget that the standard deviation/variance must be strictly positive!
        #HINT: You might want to use torch.split(): https://pytorch.org/docs/stable/generated/torch.split.html
        x      = torch.nn.LeakyReLU()(self.fc1(x))
        x      = self.fc2(x)
        mu,std = ??
        std    = ??
        return mu, std


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders.ipynb".
The first task is:
In the next cell we set the training parameters, define the loss function and create DataLoaders. Pytorch DataLoaders manage the data loading for us (break the dataset into batches, keep track of epochs, reshuffle the data after each epoch)
Can you write Python code for it?

BATCHSIZE       = 128
BATCHSIZE_TEST  = 256
LEARNING_RATE   = 1e-3

# MeanSquaredError (L2) Loss
loss_fn         = nn.MSELoss()
# Adam Optimizer
optimizer       = torch.optim.Adam(AE.parameters(), lr=LEARNING_RATE)

# Dataloaders
train_dataloader = DataLoader(training_data, batch_size=BATCHSIZE, shuffle=True)
test_dataloader  = DataLoader(test_data, batch_size=BATCHSIZE_TEST, shuffle=True)


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders.ipynb".
The first task is:
It's finally time for training:
Can you write Python code for it?

EPOCHS = 30
SEED   = 555

train_loss = []
test_loss  = []
for t in range(EPOCHS):
    torch.manual_seed(SEED)
    torch.cuda.manual_seed(SEED)
    np.random.seed(SEED)
    print(f"Epoch {t+1}\n-------------------------------")
    train_loss.append(train_loop(train_dataloader, AE, loss_fn, optimizer))
    test_loss.append(test_loop(test_dataloader, AE, loss_fn))
print("Done!")


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders.ipynb".
The first task is:
Let's see how the model is doing. Let's look at

Training and test loss
Final reconstruction quality
Can you write Python code for it?

# losses
length = len(np.asarray(train_loss).flatten())
plt.figure()
plt.plot(np.linspace(0,length*100,length), np.asarray(train_loss).flatten(),label='training set')
plt.plot(np.linspace(100,(length)*100,len(test_loss)),test_loss,label='test set')
plt.xlabel('training step')
plt.ylabel('loss')
plt.legend()
plt.show()

### zoom in
length = len(np.asarray(train_loss).flatten())
plt.figure()
plt.plot(np.linspace(0,length*100,length), np.asarray(train_loss).flatten(),label='training set')
plt.plot(np.linspace(100,(length)*100,len(test_loss)),test_loss,label='test set')
plt.xlabel('training step')
plt.ylabel('loss')
plt.legend()
plt.ylim(0,0.1)
plt.show()


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders.ipynb".
The first task is:
Let's train!
Note: Training a VAE can be unstable. If things get weird, try changing the random seed or hyperparameters, such as the number of epochs, batchsize, learning rate, sample size, likelihood noise level...
HINT: To get good artificial data samples, you need to have a KL divergence in the single digits
Can you write Python code for it?

EPOCHS = 8
SEED   = 1234

train_loss = []
test_loss  = []
for t in range(EPOCHS):
    torch.manual_seed(SEED)
    torch.cuda.manual_seed(SEED)
    np.random.seed(SEED)
    print(f"Epoch {t+1}\n-------------------------------")
    train_loss.append(train_loop(train_dataloader, VAE, negative_ELBO, optimizer))
    test_loss.append(test_loop(test_dataloader, VAE, negative_ELBO))
print("Done!")

test_loss = np.asarray(test_loss)

#TASK: plot the training loss, test loss, and the contributions to the loss from each of the two terms
length = len(np.asarray(train_loss).flatten())
plt.figure()
??
plt.xlabel('training step')
plt.ylabel('loss')
plt.legend()
plt.show()

# TASK: Inspect how the contribution of the kl divergence and log likelihood to the loss change as you change the noise in the likelihood. Some suggested values: sigma=[0.5,1,2]
# TASK: what happens when you change the number of samples?
# What do you observe? Can you interpret it?


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders.ipynb".
The first task is:
Next we create pytorch datasets from the training and test data (note that you need to change the root_dir, if you placed the data in a different folder)
Can you write Python code for it?

class SDSS_DR16(Dataset):
    """De-redshifted and downsampled spectra from SDSS-BOSS DR16"""

    def __init__(self, root_dir='drive/MyDrive/ML_lecture_data/', transform=True, train=True):
        """
        Args:
            root_dir (string): Directory of data file
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """

        if train:
            self.data = np.load(open(os.path.join(root_dir,'DR16_denoised_inpainted_train.npy'),'rb'),allow_pickle=True)
        else:
            self.data = np.load(open(os.path.join(root_dir,'DR16_denoised_inpainted_test.npy'),'rb'),allow_pickle=True)

        self.data = torch.as_tensor(self.data)
        self.mean = torch.mean(self.data)
        self.std  = torch.std(self.data)

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()

        sample = (self.data[idx]-self.mean)/self.std

        return sample

#initialize datasets
training_data = SDSS_DR16(train=True)
test_data     = SDSS_DR16(train=False)


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders.ipynb".
The first task is:
Our dataset
In this coding exercise we will be working with a galaxy spectra sample from the SDSS-BOSS DR 16 release. The spectra have been de-redshifted to the restframe and their magnitude has been standardized to a distance correspondingg to $z_\lambda=0.1$. They have further been downsampled to 1000 pixels, denoised and inpainted where masks were present.
(I can tell you more about the data cuts and preprocessing if you are interested, but it is not relevant for this task.)
Despite being relatively high-dimensional ($d=1000$), galaxy specrtra actually reside on a lower dimensional manifold. An indication for this is that we can compress them to much smaller dimensionality without sacrificing much reconstruction quality.
This property makes them a very suitable data type for VAEs.
(The same applies to image data, but images datasets are computationally more expensive to train on and they need more complicated nework architectures - things that we don't want to worry about in this exercise.)
STEP 1: Download the training and test datasets

training set
test set

and place them in your Google Drive. (If you want to avoid having to modify the file paths in the code below, you need to create a folder called 'ML_lecture_data' and place the files in there.)
Next, we link Google Drive to this notebook
Can you write Python code for it?

from google.colab import drive
import os
drive.mount('/content/drive')


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders.ipynb".
The first task is:
STEP 2: Modify the decoder network.
We will leave our decoder network as it is :)
Can you write Python code for it?

class VAEDecoder(nn.Module):

    def __init__(self, seed=620):
        super(VAEDecoder, self).__init__()
        self.seed = torch.manual_seed(seed)
        self.fc1 = nn.Linear(LATENT_SIZE,50)
        self.fc2 = nn.Linear(50,INPUT_SIZE)

    def forward(self, z):
        z = torch.nn.LeakyReLU()(self.fc1(z))
        x = self.fc2(z)
        return x


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders.ipynb".
The first task is:
STEP 3: Modify the AE class into a VAE class!

A VAE has a few more input parameters than an AE. We need a sample size, which determines how many samples we draw from $q_\phi(z|x)$ for evaluating the ELBO. We also need a $\sigma_\epsilon$ to characterize the likelihood, $p_\psi(x|z)=\mathcal{G}(x',\sigma_\epsilon)$.
The prior is fixed. We can define it in the beginning, when we initialize the VAE.
We need methods to compute the variational posterior, the likelihood, the KL-divergence and the ELBO.
Can you write Python code for it?

class VariationalAutoencoder(nn.Module):
    #TASK: add parameters mentioned in point 1.
    def __init__(self, ??, ??):
        super(VariationalAutoencoder, self).__init__()
        self.encoder = VAEEncoder()
        self.decoder = VAEDecoder()
        self."insert name of param1 here" = ??
        self."insert name of param2 here" = ??
        #TASK: Use the Normal class to define the prior (a standard normal distribution), p(z)
        self.prior       = ??

    def get_q(self,x):
        #TASK: write a method that computes q(z,x)
        #HINT: use the Normal class we imported above
        mu, std = ??
        self.q  = ??
        return True

    def sample_q(self):
        #TASK: write a method that samples from q
        #HINT: use rsample to apply the reparameterization trick
        #HINT: rsample takes a list or array of shapes as input, e.g. [sample_size]
        z_sample = ??
        return z_sample

        #TASK (Optional): write a method that allows to change how many samples are drawn from q(z|x)
    def change_sample_size(self,??):
        ?? = ??
        return True

    def get_avg_log_likelihood(self,recons,x):
        #TASK: Write a method that returns the first term in the ELBO (this method should define the likelihood and evaluate the average log likelihood of the reconstruction)
        #HINT: Pay attention to shapes. The function should return an average log likelihood (a single number) for every data point in the batch.
        #HINT: The output shape of Normal(mu, sigma).log_prob() is a little unintuitive. If mu or sigma are N-dimensional, it returns N results (applies N independent Gaussians).
        #HINT: You need to average over samples from q to obtain the final result.
        ll    = ??
        log_p = ??
        log_p = ??
        return ??

    def stochastic_kl_divergence(self,z_sample):
        #TASK: Write a method that computes the kl-divergence between q(z|x) and p(z)
        #HINT: Pay attention to shape
        return ??

    def forward(self, x):
       ...


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders.ipynb".
The first task is:
STEP 4: Prepare for training
Can you write Python code for it?

#TASK: create an instance of the Variational Autoencoder with sample_size=4 and sigma=1
VAE = ??

optimizer = torch.optim.Adam(VAE.parameters(), lr=LEARNING_RATE)

#from torch.optim.lr_scheduler import StepLR
#scheduler = StepLR(optimizer, step_size=10, gamma=0.75)

#TASK: define the new loss function
def negative_ELBO(avg_log_likelihood,kl):

    negative_ELBO = ??

    return negative_ELBO

def train_loop(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    losses = []
    for batch, X in enumerate(dataloader):
        #TASK: compute the loss from the output of the VAE foward pass
        log_likelihood, kl = ??
        loss = ??

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), batch * len(X)
            losses.append(loss)
            print(f"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]")

    #scheduler.step()
    return losses

def test_loop(dataloader, model, loss_fn):
    size = len(dataloader.dataset)
    num_batches = len(dataloader)
    test_loss, nllh, kl_ = 0, 0, 0
    model.eval()
    with torch.no_grad():
        for X in dataloader:
            #TASK: in the test loop we want to keep track not only of the ELBO, but also of the two terms that contribute to the ELBO (kl diveregence and loglikelihood)
            log_likelihood, kl = ??
            test_loss += ??.item()
            nllh += ??
            kl_ += ??

    test_loss /= num_batches
    kl_ /= num_batches
    nllh /= num_batches

    print(f" Avg test loss      : {test_loss:>8f}")
    print(f" Avg KL             : {kl_:>8f}")
    print(f" Avg negative log likelihood : {nllh:>8f} \n")

    return test_loss, kl_, nllh


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders.ipynb".
The first task is:
STEP 5: Inspect the model performance
Similar to the AE, we will look at the average reconstruction quality. But in addition, we also want to know how well the kl term was minimized. We will therefore look at three things

Reconstruction quality
Scatter plots of posterior samples and prior samples. Recall that $p(z)=\int \mathcal{d}x\, p(x,z) \approx \frac{1}{N_{samples}} \sum_{x\sim p(x)} p(z|x)$.
Quality of artificial data generation
Can you write Python code for it?

 #TASK: plot the average reconstruction error of the model as a function of wavelength (similar to above). How does it compare to the Autoencoder?
 #HINT: Use the mean of $q(z|x)$ as the latent point for data x
avg_loss  = 0
VAE.eval()
with torch.no_grad():
    for X in test_dataloader:
        pred = ??
        avg_loss+=??

plt.figure()
plt.plot(wl,np.sqrt(avg_loss))
plt.ylabel('average reconstruction error')
plt.xlabel('wavelength [Ångströms]')
plt.show()

#TASK: make a corner plot of posterior samples. Does the average posterior match the prior?

import seaborn as sns
import pandas as pd

VAE.eval()
with torch.no_grad():
    for ii, X in enumerate(test_dataloader):
        VAE.get_q(X)
        prior_sample = ??
        sample       = ??
        if ii==0:
          samples       = sample
          prior_samples = prior_sample
        else:
          samples       = np.vstack([samples, sample])
          prior_samples = np.vstack([prior_samples, prior_sample])

samples       = np.reshape(samples,[-1, LATENT_SIZE])
prior_samples = np.reshape(prior_samples,[-1, LATENT_SIZE])

print(samples.shape)
print(prior_samples.shape)

data1    = pd.DataFrame()
data2    = pd.DataFrame()

for ii in range(LATENT_SIZE):
  data1['dim_%d'%ii] = samples[:,ii]
data1['source'] = 'posterior'

for ii in range(LATENT_SIZE):
  data2['dim_%d'%ii] = prior_samples[:,ii]
data2['source'] = 'prior'

data = pd.concat([data1,data2]).reset_index(drop=True)

#HINT: to get a density estimate you can set kind='kde', but you'll probably have to reduce the number of samples, KDE optimization scales pretty badly with number of samples
sns.pairplot(data,corner=True,kind='scatter', hue='source', plot_kws={'s':4})
plt.show()

#TASK: Generate artificial data: sample from the prior and foward model the sample thorugh the decoder. Do the samples look realistic? Why?/Why not?

VAE.eval()
with torch.no_grad():
  samples      = ??
  data_samples = ??

# This is the mapping from pixel to the de-redshifted (rest) wavelength
wlmin, wlmax      =...


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders.ipynb".
The first task is:
Since we work with probability distributions in the VAE, we need to import the torch distributions package. We will only need the normal distribution for this exercise.
Can you write Python code for it?

#TASK: Familiarize yourself with torch.distribution.Normal - you can find the documentation here: https://pytorch.org/docs/stable/distributions.html#normal
#HINT: It takes a standard deviation (scale) not a variance as input
from torch.distributions import Normal as Normal


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders_solutions.ipynb".
The first task is:
... and the average reconstruction error as a function of wavelength:
Can you write Python code for it?

avg_loss  = 0
with torch.no_grad():
    for X in test_dataloader:
        pred = AE(X)
        avg_loss+=np.mean((pred.cpu().numpy()-X.cpu().numpy())**2,axis=0)/(len(test_data)//BATCHSIZE_TEST)

plt.figure()
plt.plot(wl,np.sqrt(avg_loss))
plt.ylabel('average reconstruction error')
plt.xlabel('wavelength [Ångströms]')
plt.show()

#Optional: save the model weights
#torch.save(AE.state_dict(), 'drive/MyDrive/ML_lecture_models/AE_model_weights.pth')


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders_solutions.ipynb".
The first task is:
A few things you might have noticed and that can be useful to keep in mind:

The model expects a single precision input. You can change the type of a tensor with tensor_name.type(), where tensor_name is the name of your tensor and type is the dtype. For typecasting into single precision floating points, use float(). A numpy array is typecasted with array_name.astype(type). For single precision, the type should be np.float32.
Before we analyze tensors we often want to convert them to numpy arrays with tensor_name.numpy()
If pytorch has been tracking operations that resulted in the current tensor value, you need to detach the tensor from the graph before you can transform it into a numpy array: tensor_name.detach(). Scalars can be detached with scalar.item()
If you tensor is currently on the GPU, you can bring it onto the CPU with tensor_name.cpu()

Variational Autoencoder
Your task today is to transform the above Autoencoder into a Variational Autoencoder. You can follow either follow my Step-by-Step instructions or do it your way (there's more than one solution).
We start by modifying the encoder network. Tasks are marked with the keyword #TASK, hints are marked with the keyword #HINT.
STEP 1: Modify the encoder network.
The encoder network is used to characterize the variational distribution q(z|x). Recall that we want q(z|x) to be a Gaussian with diagonal covariance. An N-dimensional Gaussian with diagonal covariance is eqivalent to N independent 1-dimensional Gaussians (where N is the latent size). Each Gaussian is defined by two quantities, its mean and variance (or, if we take the sqrt, the standard deviation).
Can you write Python code for it?

class VAEEncoder(nn.Module):

    def __init__(self, seed=853):
        super(VAEEncoder, self).__init__()
        # TASK: change the output size of the encoder network. How many parameters must it return to define q(z|x)?
        self.seed = torch.manual_seed(seed)
        self.fc1 = nn.Linear(INPUT_SIZE,50)
        self.fc2 = nn.Linear(50,LATENT_SIZE*2)

    def forward(self, x):
        # TASK: change the output of the encoder network. Instead of just returning z, it should return z and ...?
        # HINT: Don't forget that the standard deviation/variance must be strictly positive!
        # HINT: You might want to use torch.split(): https://pytorch.org/docs/stable/generated/torch.split.html
        x      = torch.nn.LeakyReLU()(self.fc1(x))
        x      = self.fc2(x)
        mu,std = torch.split(x, LATENT_SIZE,dim=-1)
        std    = torch.exp(std) + 1e-8
        return mu, std


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders_solutions.ipynb".
The first task is:
Autoencoder
This exercise starts with an Autoencoder, which is already implemented and working. The next cells walk you through the code. Your task (further below) will be to take that code and modify it into a Variational Autoencoder.
First, we define our encoder and decoder networks. We use a very simple MLP, with two linear layers and one non-linear activation function.
Can you write Python code for it?

# we inherit from pytorch Module class; https://pytorch.org/docs/stable/generated/torch.nn.Module.html

class Encoder(nn.Module):
    def __init__(self, seed=853):
        """
        seed: int, random seed for reproducibility
        """
        super(Encoder, self).__init__()
        self.seed = torch.manual_seed(seed)
        # here we are initializing the linear layers. This registeres the layer parameters (W,b) as parameters of the Module
        self.fc1 = nn.Linear(INPUT_SIZE,50)
        self.fc2 = nn.Linear(50,LATENT_SIZE)

    # this defines a forward pass of the network (="applying" the network to some input data)
    def forward(self, x):
        x      = torch.nn.LeakyReLU()(self.fc1(x))
        z      = self.fc2(x)
        return z

class Decoder(nn.Module):
    def __init__(self, seed=620):
        """
        seed: int, random seed for reproducibility
        """
        super(Decoder, self).__init__()
        self.seed = torch.manual_seed(seed)
        self.fc1 = nn.Linear(LATENT_SIZE,50)
        self.fc2 = nn.Linear(50,INPUT_SIZE)

    def forward(self, z):
        z = torch.nn.LeakyReLU()(self.fc1(z))
        x = self.fc2(z)
        return x


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders_solutions.ipynb".
The first task is:
In the next cell we set the training parameters, define the loss function and create DataLoaders. Pytorch DataLoaders manage the data loading for us (break the dataset into batches, keep track of epochs, reshuffle the data after each epoch)
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders_solutions.ipynb".
The first task is:
It's finally time for training:
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders_solutions.ipynb".
The first task is:
Let's see how the model is doing. Let's look at

Training and test loss
Final reconstruction quality
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders_solutions.ipynb".
The first task is:
Let's set some immutable variables:
The dimensionality of the input data and the dimensionality of the latent (encoded) space
Can you write Python code for it?

INPUT_SIZE      = 1000
LATENT_SIZE     = 6


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders_solutions.ipynb".
The first task is:
Our dataset
In this coding exercise we will be working with a galaxy spectra sample from the SDSS-BOSS DR 16 release. The spectra have been de-redshifted to the restframe and their magnitude has been standardized to a distance correspondingg to $z_\lambda=0.1$. They have further been downsampled to 1000 pixels, denoised and inpainted where masks were present.
(I can tell you more about the data cuts and preprocessing if you are interested, but it is not relevant for this task.)
Despite being relatively high-dimensional ($d=1000$), galaxy specrtra actually reside on a lower dimensional manifold. An indication for this is that we can compress them to much smaller dimensionality without sacrificing much reconstruction quality.
This property makes them a very suitable data type for VAEs.
(The same applies to image data, but images datasets are computationally more expensive to train on and they need more complicated nework architectures - things that we don't want to worry about in this exercise.)
STEP 1: Download the training and test datasets

training set
test set

and place them in your Google Drive. (If you want to avoid having to modify the file paths in the code below, you need to create a folder called 'ML_lecture_data' and place the files in there.)
Next, we link Google Drive to this notebook
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders_solutions.ipynb".
The first task is:
STEP 2: Modify the decoder network.
We will leave our decoder network as it is :)
Can you write Python code for it?


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders_solutions.ipynb".
The first task is:
STEP 3: Modify the AE class into a VAE class!

A VAE has a few more input parameters than an AE. We need a sample size, which determines how many samples we draw from $q_\phi(z|x)$ for evaluating the ELBO. We also need a $\sigma_\epsilon$ to characterize the likelihood, $p_\psi(x|z)=\mathcal{G}(x',\sigma_\epsilon)$.
The prior is fixed. We can define it in the beginning, when we initialize the VAE.
We need methods to compute the variational posterior, the likelihood, the KL-divergence and the ELBO.
Can you write Python code for it?

class VariationalAutoencoder(nn.Module):
    #TASK: add parameters mentioned in point 1.
    def __init__(self, sample_size, sigma):
        super(VariationalAutoencoder, self).__init__()
        self.encoder = VAEEncoder()
        self.decoder = VAEDecoder()
        self.sample_size = sample_size
        self.sigma       = sigma
        #TASK: Use the Normal class to define the prior (a standard normal distribution), p(z)
        self.prior       = Normal(torch.zeros(LATENT_SIZE), torch.ones(LATENT_SIZE))

    def change_sample_size(self,sample_size):
        self.sample_size = sample_size
        return True

    def get_q(self,x):
        #TASK: write a method that computes q(z,x)
        #HINT: use the Normal class we imported above
        mu, std = self.encoder(x)
        self.q = Normal(mu, std)
        return True

    def sample_q(self):
        #TASK: write a method that samples from q
        #HINT: use rsample to apply the reparameterization trick
        z_sample = self.q.rsample(torch.Size([self.sample_size]))
        return z_sample

    def get_avg_log_likelihood(self,recons,x):
        #TASK: Write a method that returns the first term in the ELBO (this method should define the likelihood and evaluate the average log likelihood of the reconstruction)
        #HINT: Pay attention to shapes. The function should return an average log likelihood (a single number) for every data point in the batch.
        #HINT: The output shape of Normal(mu, sigma).log_prob() is a little unintuitive. If mu or sigma are N-dimensional, it returns N results (applies N independent Gaussians).
        #HINT: You need to average over samples from q to obtain the final result.
        ll    = Normal(x[None,:,:], self.sigma)
        log_p = ll.log_prob(recons)
        log_p = torch.sum(log_p,dim=-1)
        return torch.mean(log_p,dim=0)

    def stochastic_kl_divergence(self,z_sample):
        #TASK: Write a method that computes the kl-divergence between q(z|x) and p(z)
        #HINT: Pay attention to shape
       ...


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders_solutions.ipynb".
The first task is:
STEP 4: Prepare for training
Can you write Python code for it?

#TASK: create an instance of the Variational Autoencoder with sample_size=4 and sigma=1
VAE = VariationalAutoencoder(4,sigma=1)

optimizer = torch.optim.Adam(VAE.parameters(), lr=LEARNING_RATE)

#from torch.optim.lr_scheduler import StepLR
#scheduler = StepLR(optimizer, step_size=10, gamma=0.75)

#TASK: define the new loss function
def negative_ELBO(avg_log_likelihood,kl):

    negative_ELBO = - torch.mean(avg_log_likelihood-kl)

    return negative_ELBO

def train_loop(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    losses = []
    for batch, X in enumerate(dataloader):
        #TASK: compute the loss from the output of the VAE foward pass
        log_likelihood, kl = model(X)
        loss = loss_fn(log_likelihood,kl)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), batch * len(X)
            losses.append(loss)
            print(f"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]")

    #scheduler.step()
    return losses

def test_loop(dataloader, model, loss_fn):
    size = len(dataloader.dataset)
    num_batches = len(dataloader)
    test_loss, nllh, kl_ = 0, 0, 0
    model.eval()
    with torch.no_grad():
        for X in dataloader:
            #TASK: in the test loop we want to keep track not only of the ELBO, but also of the two terms that contribute to the ELBO (kl diveregence and loglikelihood)
            log_likelihood, kl = model(X)
            test_loss += loss_fn(log_likelihood,kl).item()
            nllh += -np.mean(log_likelihood.cpu().numpy())
            kl_ += np.mean(kl.cpu().numpy())

    test_loss /= num_batches
    kl_ /= num_batches
    nllh /= num_batches

    print(f" Avg test loss      : {test_loss:>8f}")
    print(f" Avg KL             : {kl_:>8f}")
    print(f" Avg negative log likelihood : {nllh:>8f} \n")

    return test_loss, kl_, nllh


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders_solutions.ipynb".
The first task is:
STEP 5: Inspect the model performance
Similar to the AE, we will look at the average reconstruction quality. But in addition, we also want to know how well the kl term was minimized. We will therefore look at three things

Reconstruction quality
Scatter plots of posterior samples and prior samples. Recall that $p(z)=\int \mathcal{d}x\, p(x,z) \approx \frac{1}{N_{samples}} \sum_{x\sim p(x)} p(z|x)$.
Quality of artificial data generation
Can you write Python code for it?

 #TASK: plot the average reconstruction error of the model as a function of wavelength (similar to above). How does it compare to the Autoencoder?
 #HINT: Use the mean of $q(z|x)$ as the latent point for data x
avg_loss  = 0
VAE.eval()
with torch.no_grad():
    for X in test_dataloader:
        pred = VAE.decoder(VAE.encoder(X)[0])
        avg_loss+=np.mean((pred.cpu().numpy()-X.cpu().numpy())**2,axis=0)/(len(test_data)//BATCHSIZE_TEST)

plt.figure()
plt.plot(wl,np.sqrt(avg_loss))
plt.ylabel('average reconstruction error')
plt.xlabel('wavelength [Ångströms]')
plt.show()

#TASK: make a corner plot of posterior samples. Does the average posterior match the prior?

import seaborn as sns
import pandas as pd

VAE.eval()
with torch.no_grad():
    for ii, X in enumerate(test_dataloader):
        VAE.get_q(X)
        prior_sample = VAE.prior.sample([BATCHSIZE_TEST])
        sample       = VAE.sample_q().cpu().numpy()[0:1].swapaxes(0,1)
        if ii==0:
          samples       = sample
          prior_samples = prior_sample
        else:
          samples       = np.vstack([samples, sample])
          prior_samples = np.vstack([prior_samples, prior_sample])

samples       = np.reshape(samples,[-1, LATENT_SIZE])
prior_samples = np.reshape(prior_samples,[-1, LATENT_SIZE])

print(samples.shape)
print(prior_samples.shape)

data1    = pd.DataFrame()
data2    = pd.DataFrame()

for ii in range(LATENT_SIZE):
  data1['dim_%d'%ii] = samples[:,ii]
data1['source'] = 'posterior'

for ii in range(LATENT_SIZE):
  data2['dim_%d'%ii] = prior_samples[:,ii]
data2['source'] = 'prior'

data = pd.concat([data1,data2]).reset_index(drop=True)

#HINT: to get a density estimate you can set kind='kde', but you'll probably have to reduce the number of samples, KDE optimization scales pretty badly with number of samples
sns.pairplot(data,corner=True,kind='scatter', hue='source', plot_kws={'s':4})
plt.show()

#TASK: Generate artificial data: sample from the prior and foward model the sample thorugh the decoder. Do the samples look realistic?...


I am working on the file "Sessions/Session14/Day3/IntroductionToVariationalAutoencoders_solutions.ipynb".
The first task is:
Use this line to confirm the location of your files
Can you write Python code for it?


I am working on the file "Sets and Booleans.ipynb".
The first task is:
Note the curly brackets. This does not indicate a dictionary! Although you can draw analogies as a set being a dictionary with only keys.
We know that a set has only unique entries. So what happens when we try to add something that is already in a set?
Can you write Python code for it?

# Add a different element
x.add(2)

#Show
x

# Try to add the same element
x.add(1)

#Show
x


I am working on the file "Sets and Booleans.ipynb".
The first task is:
Notice how it won't place another 1 there. That's because a set is only concerned with unique elements! We can cast a list with multiple repeat elements to a set to get the unique elements. For example:
Can you write Python code for it?

# Create a list with repeats
l = [1,1,2,2,3,4,5,6,1,1]

# Cast as set to get unique values
set(l)


I am working on the file "Sets and Booleans.ipynb".
The first task is:
We can also use comparison operators to create booleans. We will go over all the comparison operators later on in the course.
Can you write Python code for it?


I am working on the file "Sets and Booleans.ipynb".
The first task is:
We can use None as a placeholder for an object that we don't want to reassign yet:
Can you write Python code for it?


I am working on the file "Sets.ipynb".
The first task is:
Mutable and Immutable Sets
Although some of the mutable elements (like list) are not allowed in set, set itself is mutable.
Can you write Python code for it?

# Mutable Set
set6 = set(['A','B','C','D','E'])
set6.add('F')
print(set6)

# Immutable Set
set7 = frozenset(['A','B','C'])
set7.add('D') # will generate error
set7

# Sets are unordered, to get ordered set use sorted function

set8 = sorted(set6)
set8


I am working on the file "Sets.ipynb".
The first task is:
Set Operations
Add, Remove Elements
Can you write Python code for it?

set6.clear()
set6

print(set4)
set4.remove('A') # will report an error if element is not present
print(set4)
set4.discard('A') # will not report an error if element is not present
print(set4)
set4.remove('A') # this will create an error

print(len(set4))

set4.add('A')
set4.add(('D','E'))
print(set4)

# add will not take any mutable object like list
set4.add(['1','2'])
set4 # this will create an error

# Union Operator
colorsA = {'Blue','Black','White','Red','Green'}
colorsB = {'Blue','White'}
colorsA.union(colorsB)

# Intersection operator
colorsA.intersection(colorsB)

x = {"a","b","c","d","e"}
x.pop()

x.pop()

# Difference between sets
x = {"a","b","c","d","e"}
y = {"b","c"}
z = {"c","d"}
x.difference(y)

x.difference(y).difference(z)

# instead of using the method difference, we can use the operator "-"
print(x - y)
print(x - y - z)

# difference_update removes all elements of another set from this set.
x = {"a","b","c","d","e"}
y = {"b","c"}
x.difference_update(y)
print(x)

# x.difference_update(y) is the same as "x = x - y"
x = {"a","b","c","d","e"}
y = {"b","c"}
x = x - y
print(x)


I am working on the file "Sets.ipynb".
The first task is:
Set can contain only unique entries. If you try to duplicate an entry in the set
Can you write Python code for it?

set3 = set()
set3.add('Blue')
set3.add('Green')
print(set3)
set3.add('Blue')
print(set3)

# please note set is case-sensitive
colors = ['Blue','Green','Red','White','blue','Blue','White','Black']
unique_colors = set(colors)
print(unique_colors)


I am working on the file "Sets.ipynb".
The first task is:
Sets will not allow any mutable objects. As you can see tuples are accepted as elements in a set while list is not which is a mutable object.
Can you write Python code for it?

set4 = {'A',('B','C')}
set4

set5 = {'X',['Y','Z']} # will generate error
set5


I am working on the file "Setup/DX_Imports.ipynb".
The first task is:
Auxilary function to map a data-vector from probes onto CpG Islands

I've got some globals going on in here
If I keep it, I'm probalby going to have to move this to a class
Can you write Python code for it?

isl = islands.sort(['Islands_Name','Relation_to_Island']).dropna()
isl = isl[isl.Islands_Name.isin(ti(isl.Islands_Name.value_counts() > 7))]

ot = other.Regulatory_Feature_Name
ot = ot[ot.isin(ti(ot.value_counts()> 7))]

gb = pd.concat([isl, probe_annotations], axis=1)
gb = gb[gb.Gene_Symbol.notnull() & gb.Islands_Name.notnull()]

g2 = gb.sort('Islands_Name')

top_gene = lambda s: s.Gene_Symbol.value_counts().index[0]
island_to_gene = g2.groupby('Islands_Name').apply(top_gene)

def map_to_islands(s):
    '''
    s is a Series of measurments on the probe level.
    '''
    on_island = s.groupby(isl.Islands_Name).mean().order()

    v = pd.concat([island_to_gene, on_island], axis=1).set_index(0)[1]
    islands_mapped_to_genes = v.groupby(level=0).mean().order()
    return on_island, islands_mapped_to_genes


I am working on the file "Setup/DX_Imports.ipynb".
The first task is:
Couple of minor tweaks
Can you write Python code for it?

def boxplot_panel(hit_vec, response_df):
    """
    Draws a series of paired boxplots with the rows of the response_df
    split according to hit_vec.
    """
    b = response_df.copy()
    #b.columns = pd.MultiIndex.from_arrays([b.columns, hit_vec.ix[b.columns]])
    b = b.T
    v1, v2 = hit_vec.unique()
    test = lambda v: Stats.anova(hit_vec, v)
    res = b.apply(test).T
    #p = res.p.order()
    p = res.p
    b = b.ix[:, p.index]

    l1 = list(b.ix[ti(hit_vec == v1)].as_matrix().T)
    l2 = list(b.ix[ti(hit_vec == v2)].as_matrix().T)

    boxes = [x for t in zip(l1, l2) for x in t]
    ax1, bp = paired_boxplot(boxes)

    y_lim = (response_df.T.quantile(.9).max()) * 1.2
    pts = [(i * 3.5 + .5, y_lim) for i, n in enumerate(p) if n < .0000001]
    if len(pts) > 0:
        s1 = ax1.scatter(*zip(*pts), marker='$**$', label='$p<10^{-5}$', s=200)
    else:
        s1 = None
    pts = [(i * 3.5 + .5, y_lim) for i, n in enumerate(p) if (n < .01)
           and (n > .0000001)]
    if len(pts) > 0:
        s2 = ax1.scatter(*zip(*pts), marker='$*$', label='$p<10^{-2}$', s=30)
    else:
        s2 = None
    #ax1.set_xticklabels(b.columns)
    ax1.set_xticklabels('')
    #ax1.set_ybound(-.2,.3)
    #ax1.legend(bp['boxes'][:2] + [s2, s1],
    #           (v1, v2, '$p<10^{-2}$', '$p<10^{-5}$'),
    #           loc='best', scatterpoints=1)


I am working on the file "Setup/DX_Imports.ipynb".
The first task is:
Create annotation probe sets
Can you write Python code for it?

cpg_island = isl.Relation_to_Island == 'Island'
dhs_site = other.DHS == 'TRUE'
enhancer = other.Enhancer == 'TRUE'
gene_body = other.UCSC_RefGene_Group.str.contains('Body')
gene_tss = other.UCSC_RefGene_Group.str.contains('TSS')
promoter = other.Regulatory_Feature_Group.str.contains('Promoter_Associated')


I am working on the file "Setup/DX_Imports.ipynb".
The first task is:
Helper for making CpG island plots
Can you write Python code for it?

def island_plot_maker(df, split, islands, ann, colors=None):
    '''
    df:      a DataFrame of probe beta values
    islands: a DataFrame mapping probes to CpG islands and
             annotations
    ann:     a DataFrame mapping probes to gene annotations
             and genomic coordinates of probe
    '''
    if colors is None:
        colors = colors_st
    groups = split.dropna().unique()
    assert len(groups) == 2

    def f(region):
        p = ti(islands.Islands_Name == region)
        p3 = ann.ix[p].join(islands.ix[p]).sort('Genomic_Coordinate')
        p = p3.index
        in_island = ti(p3.Relation_to_Island == 'Island')

        fig, ax = subplots(figsize=(10,4))
        for i,g in enumerate(groups):
            ax.scatter(p3.Genomic_Coordinate, df[ti(split == g)].ix[p].mean(1),
                       color=colors[i], label=g)
        ax.axvspan(p3.Genomic_Coordinate.ix[in_island[0]] - 30,
                   p3.Genomic_Coordinate.ix[in_island[-1]] + 30,
                   alpha=.2, color=colors[2], label='Island')
        ax.set_xlabel('Genomic Coordinate')
        ax.set_ylabel('Beta Value')
        ax.legend(loc='lower right', fancybox=True)
        prettify_ax(ax)
    return f


I am working on the file "Setup/DX_Imports.ipynb".
The first task is:
Helper functions for diffential expression
Can you write Python code for it?

def ttest_df(split_vec, df):
    dmean = df.T.groupby(split_vec).mean().T
    dvar = df.T.groupby(split_vec).var().T
    dn = df.T.groupby(split_vec).count().astype(float).T
    s12 = ((((dn[True] - 1)*dvar[True]) + ((dn[False] - 1)*dvar[False])) /
           (dn.sum(1) - 2)) ** .5
    t = (dmean[True] - dmean[False]) / (s12 * np.sqrt((1/dn[True]) + (1/dn[False])))
    t = t.dropna()
    return t

def effect_size(split_vec, df):
    dmean = df.T.groupby(split_vec).mean().T
    return pd.concat([dmean[True], dmean[False], dmean[True] - dmean[False]],
                     axis=1, keys=['g1','g2','d'])


I am working on the file "Setup/DX_Imports.ipynb".
The first task is:
Read in Gene Sets

I don't like this thing with the paths, hopfully fix later
Can you write Python code for it?

from Data.Annotations import read_in_pathways

gs, gl = read_in_pathways('/cellar/users/agross/TCGA_Code/TCGA/Extra_Data/c2.cp.v3.0.symbols_edit.csv')
gs = pd.DataFrame({p: pd.Series(1, index=s) for p,s in gs.iteritems()})
gs = gs.ix[gl.keys()].fillna(0)
gene_sets = gs


I am working on the file "Setup/DX_Imports.ipynb".
The first task is:
Read in Probe Annotations from Data-Store

These are parsed out in Compile_Probe_Annoations notebook.
Can you write Python code for it?

STORE = HDFS_DIR + 'methylation_annotation_2.h5'
islands = pd.read_hdf(STORE, 'islands')
locations = pd.read_hdf(STORE, 'locations')
other = pd.read_hdf(STORE, 'other')
snps = pd.read_hdf(STORE, 'snps')
probe_annotations = pd.read_hdf(STORE, 'probe_annotations')


I am working on the file "Setup/DX_Imports.ipynb".
The first task is:
Some additional functions I should add to my statistics module
Can you write Python code for it?

def entropy(p):
    '''
    Entropy of a methylaion vector. Here we assume 50% methylation is
    random and 0 or 1 constitute an informative measument.
    '''
    q = 1. - p
    s = np.sum((p*np.log(p)) + (q*np.log(q))) / (np.log(.5) * len(p))
    s.name = 'Entropy'
    return s


I am working on the file "Setup/DX_Imports.ipynb".
The first task is:
Some functions for dealing with entropy of a matrix
Can you write Python code for it?

def svd_entropy(df):
    U,S,vH = frame_svd(df)
    p = S ** 2 / sum(S ** 2)
    entropy = -1 * sum((p * np.log(p))) / log(len(p))
    return entropy

def entropy_gain(split, df):
    df = df.ix[:, split.index]
    h_all = svd_entropy(df)
    h_1 = svd_entropy(df.ix[:, ti(split)])
    h_0 = svd_entropy(df.ix[:, ti(split==False)])
    ratio = h_all - (h_1*split.mean() + h_0*(1-split.mean()))
    return pd.Series({'gain':ratio, 'h_all': h_all, 'h_0':h_0, 'h_1':h_1})


I am working on the file "Setup/Imports.ipynb".
The first task is:
Distribution plot with inset option
Can you write Python code for it?

def draw_dist(vec, split=None, ax=None, legend=True, colors=None,
              lim=None, tail='right', inset=False):
    """
    Draw a smooth distribution from data with an optional splitting factor.
    """
    _, ax = init_ax(ax)
    if split is None:
        split = pd.Series('s', index=vec.index)
        colors = {'s': colors} if colors is not None else None

    def get_dist(v):
        dist = smooth_dist(v)
        if tail is not None:
            dist = dist[lim:] if tail is 'right' else dist[:lim]
        return dist

    for l,v in vec.groupby(split):
        if colors is None:
            get_dist(v).plot(label=l, lw=2, ax=ax)
        else:
            get_dist(v).plot(label=l, lw=2, ax=ax, color=colors[l])
    if legend and len(split.unique()) > 1:
        ax.legend(loc='upper left', frameon=False)
    ax.set_yticks([])
    ax.set_xlabel(vec.name)
    ax.set_ylabel('probe density')

    if inset:
        a = axes([.4, .4, .5, .5])
        max_y = 0
        for l,v in vec.groupby(split):
            v = smooth_dist(v)
            if colors is None:

                a.plot(v.index, v, label=l, lw=2)
            else:
                a.plot(v.index, v, label=l, lw=2, color=colors[l])
            max_y = max(max_y, v.max())
        setp(a, yticks=[])
        a.set_xlabel(vec.name)
        a.set_ylabel('probe density')
        a.set_xlim(vec.min(), vec.max())
        a.set_ylim(0, max_y*1.03)
        a.axvspan(lim, 1, color='grey', alpha=.1)
    #ax.axvspan(0, 1, color='grey', alpha=.05)

def odds_ratio(a,b):
    ct = pd.crosstab(a,b).astype(float)
    r = (ct.ix[0,0] * ct.ix[1,1]) / (ct.ix[0,1] * ct.ix[1,0])
    return r


I am working on the file "Setup/Imports.ipynb".
The first task is:
Global Parameters
While it is not always the best practice to set global parameters, I find that for analyses such as these it is beneficial for a few reasons.
 Selection criteria can be established in one place and updates can be propagated to all subsequent analyses
 Things such as hard paths can be extracted from the logic and placed in one central location
Can you write Python code for it?

FIGDIR = '/cellar/users/agross/figures/'
HDFS_DIR = '/cellar/users/agross/Data/SSD/'

path = '/cellar/users/agross/TCGA_Code/Methlation/'
ucsd_path = path + 'data/UCSD_Methylation/'
hannum_path = path + 'data/Hannum/'
path_italy = path + 'data/EPIC_ITALY/'


I am working on the file "Setup/Imports.ipynb".
The first task is:
Manhattan plot
Can you write Python code for it?

def manhattan(vec, chrom, coords, ax=None, ybound=None,
              flow='up', ticks=True, gap=3e7):
    fig, ax = init_ax(ax, figsize=(9,3))
    x = 0
    chr_coords = []
    for i,c in enumerate(map(str, range(1,23))):
        v = vec.ix[ti(chrom == c)].dropna()
        series_scatter(coords + x, v, s=10, ax=ax,
                       color=colors[i % 5],
                       ann=None, alpha=1, rasterized=True)
        chr_len = coords.ix[v.index].max()
        x = x + chr_len + gap
        chr_coords += [x - (chr_len / 2.)]
    ax.set_xbound(gap, x + gap)
    if ybound is not None:
        ax.set_ybound(ybound[0],ybound[1])
    ax.set_xlabel('Chromosome')

    if ticks:
        ax.set_xticks(chr_coords)
        ax.set_xticklabels(map(str, range(1,23)))
    else:
        ax.set_xticks([])
    top = flow == 'down'
    prettify_ax(ax, top)


I am working on the file "Setup/Imports.ipynb".
The first task is:
Preform a logit transformation while constraining extreme values.
Can you write Python code for it?

from scipy.special import logit

logit_adj = lambda df: logit(df.clip(.001, .999))


I am working on the file "Setup/Imports.ipynb".
The first task is:
Preform two step linear adjustement.
* First on each data-source seperately
* Then on all data-sources combined
Can you write Python code for it?

def two_step_adjustment(m1, m2, age, groups):
    m1_adj = {}
    m2_adj = {}
    k = m1.index.intersection(m2.index)
    for s in groups.unique():
        pts = ti(groups == s).intersection(k)
        m1_adj[s] = detrend(age, m1.ix[pts])
        m2_adj[s] = detrend(age, m2.ix[pts])
    m1_adj = pd.concat(m1_adj.values())
    m2_adj = pd.concat(m2_adj.values())
    mc_adj = (m1_adj + m2_adj) / 2.
    mc_adj = detrend(age, mc_adj)
    mc_adj.name = 'Predicted Age (Combined)'
    return m1_adj, m2_adj, mc_adj


I am working on the file "Setup/Imports.ipynb".
The first task is:
Tweaking Display Parameters
Can you write Python code for it?

pd.set_option('precision', 3)
pd.set_option('display.width', 300)
plt.rcParams['font.size'] = 14

import seaborn as sns

sns.set_context("paper", font_scale=1.7, rc={"lines.linewidth": 2.5})
sns.set_style("white")

'''Color schemes for paper taken from http://colorbrewer2.org/'''
colors = ['#e41a1c', '#377eb8', '#4daf4a', '#984ea3', '#ff7f00',
          '#ffff33', '#a65628']
colors_st = ['#CA0020', '#F4A582', '#92C5DE', '#0571B0']
colors_th = ['#E66101', '#FDB863', '#B2ABD2', '#5E3C99']


I am working on the file "Setup/MethylationAgeModels.ipynb".
The first task is:
Horvath's Transfer Functions

He trains his model on a non-linear transform of the age as an output
These are helper functions to go back and forth the origional age and transformed age spaces
Can you write Python code for it?

def tranfer_fx(x, adult_age=20):
    x = np.float(x)
    x=(x+1)/(1+adult_age)
    y = np.log(x) if x <= 1 else x - 1
    return y

def anti_tranfer_fx(x, adult_age=20):
    if x < 0:
        return (1+adult_age)*np.exp(x)-1
    else:
        return (1+adult_age)*x+adult_age


I am working on the file "Setup/Read_HIV_Data.ipynb".
The first task is:
Adjust this data for cellular composition.  This is done after the normalization to not mess around with Horvath's pipeline too much.
Can you write Python code for it?

flow_sorted_data = pd.read_hdf(HDFS_DIR + 'methylation_annotation.h5',
                               'flow_sorted_data_horvath_norm')
cell_type = pd.read_hdf(HDFS_DIR + 'methylation_annotation.h5', 'label_map')
cell_counts = pd.read_hdf(HDFS_DIR + 'dx_methylation.h5', 'cell_counts')
cell_counts = cell_counts.groupby(level=0, axis=0).first()

n2 = flow_sorted_data.groupby(cell_type, axis=1).mean()
avg = n2[cell_counts.columns].dot(cell_counts.ix[df_hiv.columns].T)
d2 = df_hiv_n.ix[avg.index, df_hiv.columns].dropna(axis=[0,1], how='all')
cc = avg.ix[:, ti(duration=='Control')].mean(1)
df_hiv_n = (d2 - avg).add(cc, axis=0).dropna(how='all')

keepers = duration.index.intersection(df_hiv.columns.intersection(df_hiv_n.columns))
duration = duration.ix[keepers]
duration = duration.groupby(level=0).first()

consent = c1['zhang id'].isin([12373001,12805001,14055003,15455001]) == False
duration = duration.ix[duration.index.difference(ti(consent == False))]

ti(c1['zhang id'].isin([12373001,12805001,14055003,15455001]))

duration.value_counts()

store = pd.HDFStore(HDFS_DIR + 'dx_methylation.h5')
study = store['study']
age = store['age']
gender = store['gender']


I am working on the file "Setup/Read_HIV_Data.ipynb".
The first task is:
All of the patients are white or Caucasian
Can you write Python code for it?

ethnicity = clinical.ethnicity
ethnicity = ethnicity.replace('wht','white')
ethnicity = ethnicity.replace('Caucasian - European','white')
clinical['ethnicity'] = ethnicity
ethnicity.value_counts()


I am working on the file "Setup/Read_HIV_Data.ipynb".
The first task is:
As can be seen above, we have two groups of patients with respect to HIV duration, we don't really have the sample size to tease apart any differences other than this main distiction so for now I am just treating duration of HIV as a categorical variable (e.g. controls, short exposure and long exposure)
Can you write Python code for it?

duration = clinical['estimated duration hiv (months)']
duration = (1.*duration.notnull()) + (1.*duration > 100)
duration = duration.map({0:'Control',1:'HIV Short',2:'HIV Long'})
duration.value_counts()


I am working on the file "Setup/Read_HIV_Data.ipynb".
The first task is:
Batches
Can you write Python code for it?

site = clinical.Site
site.value_counts()

clinical.Utox.value_counts()


I am working on the file "Setup/Read_HIV_Data.ipynb".
The first task is:
Beck Depression Inventory is a questionarre measuring depression levels (from Wikipedia)
* 0–13: minimal depression
* 14–19: mild depression
 20–28: moderate depression
 29–63: severe depression.
Can you write Python code for it?

#Do not import
beck = clinical['beck total'].dropna()
beck.hist()

#Do not import
clinical['paofi total'].hist()


I am working on the file "Setup/Read_HIV_Data.ipynb".
The first task is:
Clean up diabetes across annotation files
Can you write Python code for it?

diabetes = clinical['diabetes'].combine_first(clinical['Diabetes @ 000'])
diabetes = diabetes.replace('N','no')
clinical['diabetes'] = diabetes
del clinical['Diabetes @ 000']
diabetes.value_counts()


I am working on the file "Setup/Read_HIV_Data.ipynb".
The first task is:
Dropping five patients because they don't look Kosher
Renormalizing cell percentages because some don't sum to 100%
Can you write Python code for it?

#Do not import
fig, axs = subplots(1,2, figsize=(9,4))
clinical.age.hist(ax=axs[0])
clinical['estimated duration hiv (months)'].hist(ax=axs[1])
axs[0].set_xlabel('Age')
axs[1].set_xlabel('estimated duration hiv (months)')
for ax in axs:
    ax.set_ylabel('# of Patients')
    prettify_ax(ax)


I am working on the file "Setup/Read_HIV_Data.ipynb".
The first task is:
Global imparement
Can you write Python code for it?


I am working on the file "Setup/Read_HIV_Data.ipynb".
The first task is:
IADL = Instrumental activities of daily living
Can you write Python code for it?

iadl = clinical.IADL
iadl.value_counts()


I am working on the file "Setup/Read_HIV_Data.ipynb".
The first task is:
Patient Selection Criteria

There are a couple of female patients in the controls, we are going to get rid of those as all of the cases are males
Most of the HIV patients are under HAART therepy, there are a few that are not and we are going to filter those out for now and possibly look at them after the primary analysis
Can you write Python code for it?

duration = duration.ix[ti(clinical['ARV Status'] != 'non-HAART')]
duration = duration.ix[ti(clinical.sex != 'F')].dropna()
duration.value_counts()


I am working on the file "Setup/Read_HIV_Data.ipynb".
The first task is:
Patient's Assessment of Own Functioning Inventory (PAOFI)
Can you write Python code for it?

#Do not import
paofi = clinical['paofi total']
paofi.hist()


I am working on the file "Setup/Read_HIV_Data.ipynb".
The first task is:
RPR (Rapid plasma reagin) is a diagnostic used to detect syphilis
Can you write Python code for it?


I am working on the file "Setup/Read_HIV_Data.ipynb".
The first task is:
Read in Clinical Data
Can you write Python code for it?

c1 = pd.read_excel(ucsd_path + 'DESIGN_Fox_v2_Samples-ChipLAyout-Clinical UNMC-UCSD methylomestudy.xlsx',
                   'HIV- samples from OldStudy', index_col=0)
c2 = pd.read_excel(ucsd_path + 'DESIGN_Fox_v2_Samples-ChipLAyout-Clinical UNMC-UCSD methylomestudy.xlsx',
                   'HIV+ samples', index_col=0)
clinical = c1.append(c2)
clinical['Sentrix_Position'] = clinical['Sentrix_Position\\'].map(lambda s: s[:-1])
del clinical['Sentrix_Position\\']


I am working on the file "Setup/Read_HIV_Data.ipynb".
The first task is:
Set up Probe Filters
Can you write Python code for it?

detection_p = pd.read_hdf(HDFS_DIR + 'dx_methylation.h5', 'detection_p')
#detection_p = detection_p[detection_p[0] > 10e-5]
detection_p = detection_p[detection_p.Sample_Name.isin(duration.index)]

ff = detection_p.groupby('level_0').size() > 3
ff.value_counts()

STORE = HDFS_DIR + 'methylation_annotation.h5'
snps = pd.read_hdf(STORE, 'snps')

snp_near = (snps.Probe_SNPs != '')
snp_near.value_counts()

probe_idx = df_hiv.index.difference(ti(ff))


I am working on the file "Setup/Read_HIV_Data.ipynb".
The first task is:
Sex is not recorded for the cases but they are all HIV+ men.
Can you write Python code for it?


I am working on the file "Setup/Read_HIV_Data.ipynb".
The first task is:
These can be broken down into 8 regimen types
Can you write Python code for it?

clinical['Regimen Type'].value_counts()

kill_list = ['zhang id', 'diabetes', 'Methylation ID',
             'Sentrix_ID','Sample_Plate','Sample_Well','Sentrix_Position',
             ]
drugs = ['ARV History', 'ARV Status', 'Current Regimen', 'Regimen Type',
         'adherence %' ,'adherent']

left = [c for c in clinical if c not in past_usage and c not in current_usage
        and c not in drugs]

age = clinical.age

clinical['BDI > 17'].value_counts()

clinical['ARV History'].value_counts()

clinical['CDC stage'].value_counts()


I am working on the file "Setup/Read_HIV_Data.ipynb".
The first task is:
We have a wide varienty of regimens with 81 unique combinations of 35 drugs
Can you write Python code for it?

reg = clinical['Current Regimen'].dropna().str.split('/').map(sorted)
drugs = {r for s in reg for r in s}
drug_mat = pd.DataFrame({i: {d: d in s for d in drugs} for i,s in
                         reg.iteritems()}).T
drug_mat.sum().order()


I am working on the file "Setup/Read_HIV_Data_supp_tables.ipynb".
The first task is:
All of the patients are white or Caucasian
Can you write Python code for it?


I am working on the file "Setup/Read_HIV_Data_supp_tables.ipynb".
The first task is:
All patients are currently using anti-retoviral therepy, but 5 patients treatment is not classified as HAART
Can you write Python code for it?

clinical['ARV History'].value_counts()

clinical['ARV Status'].value_counts()


I am working on the file "Setup/Read_HIV_Data_supp_tables.ipynb".
The first task is:
All patients are reported as adhererent, but a few are not 100% adherent
Can you write Python code for it?

clinical['adherent'].value_counts()

#Do not import
clinical['adherence %'].hist()


I am working on the file "Setup/Read_HIV_Data_supp_tables.ipynb".
The first task is:
As can be seen above, we have two groups of patients with respect to HIV duration, we don't really have the sample size to tease apart any differences other than this main distiction so for now I am just treating duration of HIV as a categorical variable (e.g. controls, short exposure and long exposure)
Can you write Python code for it?


I am working on the file "Setup/Read_HIV_Data_supp_tables.ipynb".
The first task is:
Dropping five patients because they don't look Kosher
Renormalizing cell percentages because some don't sum to 100%
Can you write Python code for it?

keepers = labs.index.diff(['RG065','RG175','RG279','RA182','RM285'])
hiv_imm[cell_types] = hiv_imm[cell_types].div(hiv_imm[cell_types].sum(1), axis=0) * 100
keep = pd.Series('Keep', keepers).ix[hiv_imm.index].fillna('Reject')
hiv_imm['QC status'] = keep
hiv_imm.to_csv(FIGDIR + 'HIV_cell_comp.csv')

#Do not import
fig, axs = subplots(1,2, figsize=(9,4))
clinical.age.hist(ax=axs[0])
clinical['estimated duration hiv (months)'].hist(ax=axs[1])
axs[0].set_xlabel('Age')
axs[1].set_xlabel('estimated duration hiv (months)')
for ax in axs:
    ax.set_ylabel('# of Patients')
    prettify_ax(ax)


I am working on the file "Setup/Read_HIV_Data_supp_tables.ipynb".
The first task is:
Fix BMI to unified labels
Can you write Python code for it?

bmi = clinical['bmi'].combine_first(clinical['BMI'])
clinical['BMI'] = bmi
clinical = clinical[clinical.columns.diff(['bmi'])]

#Do not import
bmi.hist()

current_usage = ["Current 'Other' dx", 'Current Alcohol dx',
                 'Current Bipolar I', 'Current Bipolar II',
                 'Current Cannabis dx', 'Current Cocaine dx',
                 'Current Dysthymia', 'Current Halucinogen dx',
                 'Current Inhalant dx', 'Current MDD',
                 'Current Methamphetamine dx', 'Current Opioid dx',
                 'Current PCP dx', 'Current Sedative dx',
                 'Any Current Substance dx']
current_usage = clinical[current_usage]
current_usage.dropna(how='all').apply(pd.value_counts).fillna(0).T

past_usage = ["LT 'Other' dx",  'LT Alcohol dx', 'LT Bipolar I',
              'LT Bipolar II', 'LT Cannabis dx', 'LT Cocaine dx',
              'LT Dysthymia', 'LT Halucinogen dx', 'LT Inhalant dx',
              'LT MDD', 'LT Methamphetamine dx', 'LT Opioid dx',
              'LT PCP dx', 'LT Sedative dx', 'Any LT Substance dx']
past_usage = clinical[past_usage]
past_usage.dropna(how='all').apply(pd.value_counts).fillna(0).T

usage = current_usage.join(past_usage).dropna(how='all')
usage.to_csv(FIGDIR + 'drug_usage.csv')

merge_col = ['age','BMI','sex','diabetes','ethnicity','Sample_Plate','Sample_Well',
             'Sentrix_ID','Sentrix_Position']
shared_clinical = clinical[merge_col]
shared_clinical.to_csv(FIGDIR + 'shared_clinical.csv')

c = [u'Site', u'ARV History', u'ARV Status', u'BDI > 17', u'CDC stage',
     u'estimated duration hiv (months)', u'Current Regimen',
     u'Regimen Type', u'adherence %',
     u'HCV', u'IADL', u'RPR', u'Utox',
     u'beck total', u'global impairment', u'paofi total',]
hiv_clinical = clinical[c].dropna(axis=0)
hiv_clinical.to_csv(FIGDIR + 'hiv_clinical.csv')


I am working on the file "Setup/Read_HIV_Data_supp_tables.ipynb".
The first task is:
IADL = Instrumental activities of daily living
Can you write Python code for it?


I am working on the file "Setup/Read_HIV_Data_supp_tables.ipynb".
The first task is:
Patient Selection Criteria

There are a couple of female patients in the controls, we are going to get rid of those as all of the cases are males
Most of the HIV patients are under HAART therepy, there are a few that are not and we are going to filter those out for now and possibly look at them after the primary analysis
Can you write Python code for it?


I am working on the file "Setup/Read_HIV_Data_supp_tables.ipynb".
The first task is:
Patient's Assessment of Own Functioning Inventory (PAOFI)
Can you write Python code for it?


I am working on the file "Setup/Read_HIV_Data_supp_tables.ipynb".
The first task is:
RPR (Rapid plasma reagin) is a diagnostic used to detect syphilis
Can you write Python code for it?


I am working on the file "Setup/Read_HIV_Data_supp_tables.ipynb".
The first task is:
Read in Clinical Data
Can you write Python code for it?


I am working on the file "Setup/Read_HIV_Data_supp_tables.ipynb".
The first task is:
Read in HIV Methylation data
Read in quantile-normalized data, adjusted for cellular compositions and then normalized agin using BMIQ.
Can you write Python code for it?

df_hiv = pd.read_hdf('/data/methylation_norm.h5', 'quant_BMIQ_adj')
df_hiv = df_hiv.ix[:, duration.index]
df_hiv = df_hiv.dropna(1)


I am working on the file "Setup/Read_HIV_Data_supp_tables.ipynb".
The first task is:
Sex is not recorded for the cases but they are all HIV+ men.
Can you write Python code for it?


I am working on the file "Setup/Read_HIV_Data_supp_tables.ipynb".
The first task is:
These can be broken down into 8 regimen types
Can you write Python code for it?

clinical['Regimen Type'].value_counts()

kill_list = ['zhang id', 'diabetes', 'Methylation ID', 'Diabetes @ 000',
             'Sentrix_ID','Sample_Plate','Sample_Well','Sentrix_Position\\',
             ]
drugs = ['ARV History', 'ARV Status', 'Current Regimen', 'Regimen Type',
         'adherence %' ,'adherent']

left = [c for c in clinical if c not in past_usage and c not in current_usage
        and c not in drugs]

age = clinical.age

clinical['BDI > 17'].value_counts()

clinical['ARV History'].value_counts()

clinical['CDC stage'].value_counts()


I am working on the file "Setup/Read_HIV_Data_supp_tables.ipynb".
The first task is:
Trimming the clinical dataset
None of the patients are diabetic
Can you write Python code for it?

clinical.diabetes.value_counts()

clinical['Diabetes @ 000'].value_counts()


I am working on the file "Setup/Read_HIV_Data_supp_tables.ipynb".
The first task is:
Update clinical data with new data provided by Howard Fox
Can you write Python code for it?

age_new = pd.read_csv(ucsd_path + 'UpdatesAges-Infection.csv', index_col=0)
age = age_new.age.combine_first(clinical.age)
age.name= 'age'
clinical['age'] = age
l = 'estimated duration hiv (months)'
clinical[l] = age_new['Estimated Duration HIV+ (months)'].combine_first(clinical[l])


I am working on the file "Setup/Read_HIV_Data_supp_tables.ipynb".
The first task is:
We have a wide varienty of regimens with 81 unique combinations of 35 drugs
Can you write Python code for it?


I am working on the file "Setup/clinical_figure.ipynb".
The first task is:
LLQ: limit of HIV rna quantification
APRI: liver disease progression (AST to Platelet Ratio Index)
Can you write Python code for it?

pd.crosstab(labs['LLQ PLASMA'], duration)

fisher_exact_test(duration=='HIV Long', labs['LLQ PLASMA']=='>LLQ')

c = ['LLQ PLASMA', 'PT', 'INR']

c = ['WBC','RBC','HGB','HCT','MCV','MCH','MCHC','Platelets']
fig, axs = subplots(4,2, figsize=(8,10))
axs = axs.flatten()
for i,b in enumerate(c):
    labs[b].hist(ax=axs[i], color=hiv_color)
    axs[i].set_xlabel(b)
    axs[i].set_ylabel('No. of patients')
    prettify_ax(axs[i])

letters = list(map(chr, range(97, 123)))[8:]
for i,ax in enumerate(axs.flatten()):
    ax.text(-0.15, 1.15, letters[i].upper(), transform=ax.transAxes,
            fontsize=20, fontweight='bold', va='top', ha='right')
    prettify_ax(ax)
fig.tight_layout()
fig.savefig(FIGDIR + 'sup_clinical_blood.pdf')


I am working on the file "Shallow_water.ipynb".
The first task is:
If you wish to examine the Python code for this chapter, see

exact_solvers/shallow_water.py ...
   on github,
exact_solvers/shallow_demos.py ...
   on github.

For convenience we will work in units such that the gravitational constant $g=1$.
Can you write Python code for it?


I am working on the file "Shallow_water.ipynb".
The first task is:
In the interactive notebook you can check the boxes to plot the characteristic families, and notice that the 1-characteristics impinge on the 1-shock; the 2-characteristics impinge on the 2-shock.  Thus these shocks satisfy the entropy condition.  You can also check a box to show the particle paths, which show how the water is decelerated to 0 speed as it goes through each shock.
Dam-break problem: all-shock solution
Here's another example, which you might think of as modeling a dam that suddenly breaks.  The water is initially at rest on both sides, but much deeper on the left.  We can again construct a 2-shock Riemann solution, but this is not physically correct in this case!
Can you write Python code for it?

shallow_water.plot_riemann_SW(h_l=4, h_r=1, u_l=0, u_r=0, plot1=True,
                              force_waves='shock', particle_paths=False);


I am working on the file "Shallow_water.ipynb".
The first task is:
In the interactive notebook you can confirm that 1-charactersitics spread out across the rarefaction fan while 2-characteristics converge on the shock.  View the particle paths and note that between the two waves the fluid velocity is constant, with the fluid accelerating across both the rarefaction and shock to the same intermediate value $u_m$.
Dry states
The approach just described will yield the correct Riemann solution (provided that the rootfinding algorithm converges) in most situations.  However, it will fail if the solution involves dry ($h=0$) states.
Dry middle state
Sometimes it seems that there is no way to connect a given pair of left and right states via entropy-satisfying shocks or rarefactions.  In the cell below we plot only the physically relevant parts of the integral curves and Hugoniot loci passing through the left and right states.  Both the $h-u$ and $h-hu$ planes are shown.  With the interactive widget you can try to adjust those states to find a combination for which none of the curves intersect.
Can you write Python code for it?

interact(shallow_demos.connect_states,
         h_l=widgets.FloatSlider(min=0.001,max=2,value=1),
         u_l=widgets.FloatSlider(min=-5,max=5,value=-1),
         h_r=widgets.FloatSlider(min=0.001,max=2,value=1),
         u_r=widgets.FloatSlider(min=-5,max=5,value=1));


I am working on the file "Shallow_water.ipynb".
The first task is:
Interactive solution
Finally, the notebook contains code that generates an interactive plot showing the solution to the Riemann problem for arbitrary left and right states, including solutions with dry states.
Can you write Python code for it?

def plot_exact_riemann_solution(h_l=3.,u_l=0.,h_r=1.,u_r=0.,t=0.2, fig=0):
    plot_function = shallow_water.make_demo_plot_function(h_l,h_r,u_l,u_r,
                                                          hlim=(0,6),ulim=(-3,3))
    plot_function(t,fig)

interact(plot_exact_riemann_solution,
         h_l=FloatSlider(min=0.,max=5.,step=0.1,value=3.,
                                 description=r'$h_l$'),
         u_l=FloatSlider(min=-2.5,max=2.5,step=0.1,value=0.,
                                 description=r'$u_l$'),
         h_r=FloatSlider(min=0.,max=5.,step=0.1,value=1.,
                                 description=r'$h_r$'),
         u_r=FloatSlider(min=-2.5,max=2.5,step=0.1,value=0.,
                                 description=r'$u_r$'),
         t=FloatSlider(min=0., max=0.6, step=0.1,value=0.),
         fig=fixed(0));


I am working on the file "Shallow_water.ipynb".
The first task is:
Near the point of intersection, the curves are very close; indeed, they must be tangent at this point since their direction is parallel to the corresponding eigenvector there (and in fact they also have the same curvature).  Far from this point they diverge; for small depths they must diverge greatly, since the Hugoniot locus never reaches $h=0$ at any finite velocity.
The structure of centered rarefaction waves
Rarefactions appearing in the Riemann solution are also similarity solutions; that is, they are constant along any ray $\xi=x/t$.  For a $p$-rarefaction connecting states $q_\ell$ and $q_r$, the rarefaction thus has the form
\begin{align}
    q(x,t) & = \begin{cases} q_\ell & \text{if } x/t \le \lambda_p(q_\ell), \
                            \tilde{q}(x/t) & \text{if } \lambda_p(q_\ell) \le \lambda_p(q_r), \
                            q_r & \text{if } x/t \ge \lambda_p(q_r). \end{cases}
\end{align}
It remains to be determined how $\tilde{q}$ varies as a function of $x/t$.  This can be determined by noting that each solution value $\tilde{q}$ propagates at the characteristic speed $\lambda_p(\tilde{q})$.  Letting $\xi=x/t$, this gives us the condition
\begin{align} \label{SW:char-sim}
    \lambda_p(\tilde{q}(\xi)) = \xi.
\end{align}
By combining (\ref{SW:char-sim}) with the Riemann invariant condition, we can find $\tilde{h}$ and $\tilde{u}$ within the rarefaction.  For concreteness, let us consider a 1-rarefaction connecting $q_\ell$ and $q_m$.  Then (\ref{SW:char-sim}) reads $\tilde{u} - \sqrt{g\tilde{h}} = \xi$, which simplifies to
\begin{align} \label{SW:char-sim1}
\tilde{h} = \frac{(\tilde{u}-\xi)^2}{g}.
\end{align}
Meanwhile, the fact that $w_1(\tilde{q})$ is constant means that
\begin{align} \label{SW:w1-const}
    \tilde{u} + 2 \sqrt{g\tilde{h}} = u_\ell + 2 \sqrt{gh_\ell}.
\end{align}
Combining (\ref{SW:char-sim1}) with (\ref{SW:w1-const}), we find
\begin{align} \label{SW:1-raref-struct-h}
    \tilde{h} & = \frac{(u_\ell + 2\sqrt{gh_\ell} - \xi)^2}{9g} = \frac{(w_1(q_\ell) - \xi)^2}{9g}, \
 ...
Can you write Python code for it?

shallow_water.plot_riemann_SW(h_l=1, h_r=1, u_l=-1., u_r=1.)


I am working on the file "Shallow_water.ipynb".
The first task is:
Note that in this plane the curves in each family are simply vertical translates of one another, and all curves asymptote to $\pm \infty$ as $h \rightarrow 0$.  This means that it is impossible to have a shock with an adjacent dry ($h=0$) state.
All-shock Riemann solutions
If we knew a priori that both waves in the Riemann solution were shock waves, we could find $q_m$ by finding the intersection of the 1-Hugoniot locus passing through $q_\ell$ with the 2-Hugoniot locus passing through $q_r$.  The Riemann solution would then consist simply of two discontinuities propagating at the speeds given by (\ref{SW:shock-speed}).
Here is an example for which this is the correct solution: The initial depth is the same everywhere, $h_\ell=h_r$, but the velocity $u_\ell>0$ while $u_r<0$ so the flow is toward $x=0$ from both sides.  Note that in this plot the dark and light blue stripes show a passive tracer advected with the flow, which is shown only to help visualize the fluid velocity in these plots.
Can you write Python code for it?

shallow_water.plot_riemann_SW(h_l=2, h_r=2, u_l=1, u_r=-1)


I am working on the file "Shallow_water.ipynb".
The first task is:
Note that in this plane the integral curves of each family are simply vertical translates of one another due to the form of the functions $w_1$ and $w_2$.  Unlike the Hugoniot loci, the integral curves do not asymptote to $\pm\infty$ as $h \rightarrow 0$ and instead each approaches a finite value.
Comparison of integral curves and Hugoniot loci
You may have noticed that the integral curves look very similar to the Hugoniot loci, especially when plotted in the $h-hu$ plane.  Below we emphasize the differences by plotting the curve from each of these families that passes through a specified point.  The difference is less noticeable in the $h-hu$ plane since all of the curves approach the origin.
In the notebook you can select which family to display.
Can you write Python code for it?

interact(shallow_demos.compare_curves,
         wave_family=RadioButtons(options=[1,2],
                                  description='Wave family:'),
         y_axis=RadioButtons(options=['u','hu'],
                             description='Vertical axis:'),
         h0=FloatSlider(min=1.e-1,max=3.,value=1.,
                        description='$h_*$'),
         u0=FloatSlider(min=-3,max=3,description='$u_*$'));


I am working on the file "Shallow_water.ipynb".
The first task is:
Notice that the 1-characteristics (which are plotted as thin lines) don't impinge on the 1-shock; instead, characteristics are diverging away from it.  This shock does not satisfy the entropy condition and should be replaced with a rarefaction.  The corresponding part of the Hugoniot locus is plotted with a dashed line to remind us that it is unphysical.
Now check the box to plot the 2-characteristics.  Notice that these do impinge on the 2-shock; this is an entropy-satisfying shock, and the Hugoniot locus is plotted as a solid line.
The entropy condition
For the 2-shock Riemann solution to be physically correct, each of the waves must satisfy the Lax entropy condition, with the characteristic speed to the left of the wave greater than that to the right.  Notice that the shock speed (\ref{SW:shock-speed}) can be written as
\begin{align}
    s & = u_ \pm \sqrt{gh \frac{h_ + h}{2h_*} }.
\end{align}
Thus the 1-shock entropy condition reads as
\begin{align}
u_\ell - \sqrt{gh_\ell} > u_\ell - \sqrt{gh_\ell \frac{h_\ell+h_m}{2h_\ell}} = u_m - \sqrt{gh_m \frac{h_m+h_\ell}{2h_m}} > u_m - \sqrt{gh_m}.
\end{align}
The second expression for the shock speed is obtained by noticing
that the shock speed is invariant under transposition of the left and right states.
The first and second inequalities both simplify to
\begin{align} \label{SW:1-entropy}
    h_m > h_\ell.
\end{align}
Similarly, it can be shown that the entropy condition for a 2-shock connecting $q_m$ and $q_r$ is satisfied if and only if
\begin{align} \label{SW:2-entropy}
    h_m > h_r.
\end{align}
If (\ref{SW:1-entropy}) or (\ref{SW:2-entropy}) is violated, then the
corresponding wave must be a rarefaction rather than a shock.
Go back and look at the phase plane plots above.  You'll notice that the 1-Hugoniot locus passing through $q_\ell$ is plotted with a solid line for $h>h_\ell$ and with a dashed line for $h<h_\ell$ (and the same is done for the 2-Hugoniot locus passing through $q_r$).  This is to remind us that states connected to $q_\ell$ by the dashed...
Can you write Python code for it?

interact(shallow_demos.plot_int_curves, y_axis=widgets.fixed('hu'),
         plot_1=widgets.Checkbox(description='1-wave curves',
                                 value=True),
         plot_2=widgets.Checkbox(description='2-wave curves',
                                 value=False));


I am working on the file "Shallow_water.ipynb".
The first task is:
Notice that the 2-characteristics (plotted as thin lines) impinge on the 2-rarefaction; in fact they intersect at the left edge of the rarefaction. This means that the solution we constructed is triple-valued and nonsensical as a solution to this one-dimensional conservation law, and so this portion of the solution is omitted in the plots of depth and momentum.  In this case a rarefaction wave is not physical and should be replaced with a shock; the corresponding part of the integral curve is hence shown as a dashed line.
The Riemann solution
To determine $q_m$, we need to know whether each of the two waves is a shock or rarefaction, so that we can use the appopriate Hugoniot locus or integral curve.  But to determine the nature of each wave, we need to check (\ref{SW:1-entropy}) or (\ref{SW:2-entropy}), which requires knowledge of $h_m$.  To deal with this we define a piecewise function $\phi_\ell(h)$ that gives

for $h>h_\ell$, the momentum connected to $q_\ell$ by a 1-shock;
for $h<h_\ell$, the momentum connected to $q_\ell$ by a 1-integral curve.

Thus $\phi$ is partly a 1-Hugoniot locus and partly a 1-integral curve; it gives the value of each precisely in the regime where they are physically correct.  We can similarly define a function $\phi_r(h)$ related to the 2-Hugoniot locus and 2-integral curve.  The middle state depth $h_m$ is the value for which $\phi_\ell(h) = \phi_r(h)$.  This value can be found computationally via a rootfinding algorithm.
Dam-break problem: correct solution
Below we plot (at last) the correct solution to the dam-break problem, involving a 1-rarefaction and a 2-shock.  Plot the characteristics and notice how each family behaves correctly with respect to the corresponding wave in the Riemann solution.
Can you write Python code for it?

shallow_water.plot_riemann_SW(h_l=4, h_r=1, u_l=0, u_r=0)


I am working on the file "Shallow_water.ipynb".
The first task is:
Notice that the segment of each integral curve that connects to states with a smaller depth is plotted as a solid line, while the segment connecting to states with greater depth is plotted with a dashed line.  This again is to remind us that states connected by a rarefaction through the solid part are physical (entropy-satisfying), while states connected by the dashed part would be unphysical (entropy-violating).
Try changing the velocities to $(1.9, -1.9)$, and notice that the Riemann solution has a state with almost zero depth in the middle.  What happens if you use even larger outflow velocities?  We'll discuss this later, in the section on dry states.
Dam-break problem: all-rarefaction solution
Next let us revisit the problem from the section above where we found that the all-shock solution was incorrect.  This time we'll try to find an all-rarefaction solution.
Can you write Python code for it?

shallow_water.plot_riemann_SW(h_l=4, h_r=1, u_l=0, u_r=0, plot2=True,
                              force_waves='raref', particle_paths=False)


I am working on the file "Shallow_water.ipynb".
The first task is:
We can also plot the integral curves in the $h$--$u$ plane:
Can you write Python code for it?

interact(shallow_demos.plot_int_curves, y_axis=widgets.fixed('u'),
         plot_1=widgets.Checkbox(description='1-wave curves',
                                 value=True),
         plot_2=widgets.Checkbox(description='2-wave curves',
                                 value=False));


I am working on the file "Shallow_water.ipynb".
The first task is:
You should find that by making the initial states flow sufficiently fast away from each other, there is no intersection in the $h-u$ plane.  In the $h-hu$ plane, the integral curves always intersect at the origin.  The reason for this ambiguity is that for zero depth it isn't meaningful to assign a velocity.  Thus in the $h-u$ plane we could think of the entire $u$-axis as being part of every integral curve.  That means that we can always connect the left and right states via an intermediate state with depth $h=0$ (a dry state).
Since the 1-integral curve through $q_\ell$ reaches $h=0$ at $u = u_\ell + 2 \sqrt{gh_\ell}$ and the 2-integral curve reaches $h=0$ at $u=u_r-2\sqrt{gh_r}$, we see that a dry middle state occurs if and only if $u_\ell + 2 \sqrt{gh_\ell} < u_r-2\sqrt{gh_r}$.
It is clear in this case that the solution must consist of two centered rarefaction waves, connecting the left and right states to the middle dry state.  These centered rarefactions still have the structure derived above; to complete the solution we only need to know the range of centered characteristics included in each rarefaction.  This is most conveniently determined by considering the $h-u$ plane.  The 1-integral curve passing through $q_\ell$ is given by $u = u_\ell + 2(\sqrt{gh_\ell}-\sqrt{gh_r})$; it reaches a dry state $h=0$ at $u=u_\ell + 2\sqrt{gh_\ell}$.  Similarly, the 2-integral curve passing through $q_r$ reaches a dry state $h=0$ at $u=u_r - 2\sqrt{gh_r}$.  Thus the dry state solution is given by
\begin{align}
    q(x,t) & = \begin{cases}
                        q_\ell &         \text{if } x/t \le u_\ell - \sqrt{gh_\ell}, \
                        \tilde{q}\ell & \text{if } u\ell -  \sqrt{gh_\ell} \le x/t \le u_\ell+2\sqrt{gh_\ell}, \
                        0 &           \text{if } u_\ell + 2\sqrt{gh_\ell} \le x/t \le u_r - 2\sqrt{gh_r}, \
                        \tilde{q}r & \text{if } u_r - 2\sqrt{gh_r} \le x/t \le u_r+\sqrt{gh_r}, \
                        q_r &         \text{if } x/t \ge u_r + \sqrt{gh_r},
 ...
Can you write Python code for it?

shallow_water.plot_riemann_SW(h_l=0.5, h_r=0.5, u_l=-1.9, u_r=1.9)


I am working on the file "Shallow_water_approximate.ipynb".
The first task is:
An implementation of this solver for use in Clawpack can be found here.
Examples using the HLL solver
Can you write Python code for it?

def compare_solutions(q_l, q_r, solvers):
    outputs = []
    states = {}

    for solver in solvers:
        if solver.lower() == 'exact':
            outputs.append(sw.exact_riemann_solution(q_l,q_r))
        if solver.lower() == 'hll':
            outputs.append(shallow_water_hll(q_l, q_r))
            states['hll'] = outputs[-1][0]
        if solver.lower() == 'roe':
            outputs.append(shallow_water_roe_with_efix(q_l, q_r))
            states['roe'] = outputs[-1][0]
        if solver.lower() == 'roe no efix':
            outputs.append(shallow_water_roe(q_l, q_r))
            states['roe no efix'] = outputs[-1][0]

    plot_function = \
        rt.make_plot_function([val[0] for val in outputs],
                              [val[1] for val in outputs],
                              [val[2] for val in outputs],
                              [val[3] for val in outputs],
                              solvers, layout='horizontal',
                              variable_names=sw.conserved_variables)

    interact(plot_function,
             t=widgets.FloatSlider(min=0,max=0.9,step=0.1,value=0.4));

    return states


I am working on the file "Shallow_water_approximate.ipynb".
The first task is:
An implementation of this solver for use in Clawpack can be found here.
Roe solver examples
Here are some examples of problems with transonic rarefactions, showing the behavior of the solver.  The first example involves two rarefaction waves; the left-going one is transonic.
Can you write Python code for it?

h_l = 1; h_r = 1
u_l = 0.5; u_r = 2

def compare_efix(h_l,u_l,h_r,u_r):
    hu_l = h_l*u_l; hu_r = h_r*u_r
    q_l = np.array([h_l, hu_l]); q_r = np.array([h_r, hu_r])
    ex_states, ex_speeds, ex_reval, ex_wave_types = \
                        sw.exact_riemann_solution(q_l,q_r)
    states, speeds, reval, wave_types = \
                        shallow_water_roe_with_efix(q_l, q_r)
    no_efix_states, no_efix_speeds, no_efix_reval, no_efix_wave_types = \
                        shallow_water_roe(q_l, q_r)

    plot_function = \
        rt.make_plot_function([ex_states, no_efix_states, states],
                              [ex_speeds, no_efix_speeds, speeds],
                              [ex_reval, no_efix_reval, reval],
                              [ex_wave_types, no_efix_wave_types, wave_types],
                              ['Exact','Roe w/o fix','Roe with fix'],
                              layout='vertical', vertical_spacing=0.1,
                              variable_names=sw.conserved_variables)
    interact(plot_function, t=widgets.FloatSlider(min=0,max=.9,step=.1,value=0.4));
    return q_l, q_r, states, no_efix_states

q_l, q_r, states, no_efix_states = compare_efix(h_l,u_l,h_r,u_r)

fig, ax = plt.subplots(1,2,figsize=(10,4))
sw.phase_plane_plot(q_l,q_r,g=1.,y_axis='u',
                    approx_states=no_efix_states,ax=ax[0])
sw.phase_plane_plot(q_l,q_r,g=1.,y_axis='u',
                    approx_states=states,ax=ax[1],color='orange')
ax[0].set_title('Without entropy fix')
ax[1].set_title('With entropy fix');


I am working on the file "Shallow_water_approximate.ipynb".
The first task is:
Because the exact and approximate solutions are the same, the exact solution is not visible in these plots.  The only exception is the 1-wave in the first plot; the speed of the 1-wave appears different for the Roe solution and the exact solution, but this is unimportant since the 1-wave has zero strength.
Entropy fix
As discussed previously, the use of an entropy-violating discontinuity in place of a rarefaction can be problematic in the case of a transonic rarefaction.  For the Roe solver, we can again apply an entropy fix, splitting the single discontinuity into two waves, to avoid this problem.  Several different entropy fixes have been proposed; here we focus on one that is quite general and effective in practice.
Suppose that $\lambda_1(q_\ell) < 0$ while $\lambda_1(q_m)>0$.  In this case, it appears that the 1-wave should be a transonic rarefaction.  We take the 1-wave ${\mathcal W}1$ of the Roe solver, with speed $s_1$, and split it into two waves, ${\mathcal W}{1\ell} = \alpha_{1\ell} r_1$ and ${\mathcal W}{1r} = \alpha{1r} r_1$, with speeds $\lambda_1(q_\ell), \lambda_1(q_m)$.  Two requirements determine the strength of these waves.  First, the total jump across them should be equal to the jump across the original wave:
$${\mathcal W}{1\ell} + {\mathcal W}{1r} = {\mathcal W}_1.$$
Second, we must maintain conservation:
$$\lambda_1(q_\ell) {\mathcal W}{1\ell} + \lambda_1(q_m) {\mathcal W}{1r} = s_1 {\mathcal W}_1.$$
This leads to
\begin{align}
{\mathcal W}{1\ell} & = \frac{\lambda_1(q_m)-s_1}{\lambda_1(q_m)-\lambda_1(q\ell)}{\mathcal W}1 \
{\mathcal W}{1r} & = \frac{s_1 - \lambda_1(q_\ell)}{\lambda_1(q_m)-\lambda_1(q_\ell)}{\mathcal W}_1.
\end{align}
We can check in a similar way for a transonic 2-rarefaction, and modify the 2-wave accordingly.  Note that only one of the two characteristic fields can be transonic in the Riemann solution for the shallow water equations.
This entropy fix is implemented in the solver below.
Can you write Python code for it?

def shallow_water_roe_with_efix(q_l, q_r, grav=1., efix=True):
    """
    Approximate Roe solver for the shallow water equations,
    with entropy fix.
    """

    h_l = q_l[0]
    hu_l = q_l[1]
    u_l = hu_l/h_l
    h_r = q_r[0]
    hu_r = q_r[1]
    u_r = hu_r/h_r

    delta = q_r - q_l

    # Roe averages
    h_hat = (h_r + h_l)/2.
    u_hat = (np.sqrt(h_r)*u_r + np.sqrt(h_l)*u_l) /  \
            (np.sqrt(h_r) + np.sqrt(h_l))
    c_hat = np.sqrt(grav*h_hat)

    s1 = u_hat - c_hat
    s2 = u_hat + c_hat

    alpha1 = ( (u_hat+c_hat)*delta[0] - delta[1])/(2*c_hat)
    #alpha2 = (-(u_hat-c_hat)*delta[0] + delta[1])/(2*c_hat)

    h_m = q_l[0] + alpha1
    hu_m = q_l[1] + alpha1 * (u_hat - c_hat)
    u_m = hu_m/h_m
    q_m = np.array([h_m, hu_m])

    transonic = None
    if efix:
        # Check if either wave appears transonic
        lambda_1_l = u_l - np.sqrt(grav*h_l)
        lambda_1_m = u_m - np.sqrt(grav*h_m)
        lambda_2_m = u_m + np.sqrt(grav*h_m)
        lambda_2_r = u_r + np.sqrt(grav*h_r)

        if lambda_1_l < 0 < lambda_1_m: # 1-wave appears transonic
            transonic = 1
        elif lambda_2_m < 0 < lambda_2_r: # 2-wave appears transonic
            transonic = 2

    if transonic == 1:
        beta = (lambda_1_m - s1)/(lambda_1_m - lambda_1_l)
        alpha1_l = beta*alpha1
        #alpha_1_r = (1-beta)*alpha1
        h_m_new = q_l[0] + alpha1_l
        hu_m_new = q_l[1] + alpha1_l * (u_hat-c_hat)
        q_m_new = np.array([h_m_new, hu_m_new])
        states = np.column_stack([q_l,q_m_new,q_m,q_r])
        speeds = [lambda_1_l, lambda_1_m, s2]
        wave_types = ['contact']*3

        def reval(xi):
            h_out  = (xi<speeds[0])*states[0,0] + \
                     (speeds[0]<=xi)*(xi<speeds[1])*states[0,1] + \
                     (speeds[1]<=xi)*(xi<speeds[2])*states[0,2] + \
                     (speeds[2]<=xi)*states[0,3]
            hu_out = (xi<speeds[0])*states[1,0] + \
                    ...


I am working on the file "Shallow_water_approximate.ipynb".
The first task is:
Entropy and positivity
It can be shown that the HLLE solver always preserves positivity and does not require an entropy fix.  Both of these properties result in part from the fact that the HLLE solver is more diffusive than the Roe solver.
Can you write Python code for it?

# Transonic rarefaction example b
h_l = 4; h_r = 1
u_l = 4; u_r = 1

hu_l = h_l*u_l; hu_r = h_r*u_r
q_l = np.array([h_l, hu_l]); q_r = np.array([h_r, hu_r])

states = compare_solutions(q_l, q_r, ['Exact','HLL'])

sw.phase_plane_plot(q_l,q_r,g=1.,y_axis='u',
                    approx_states=states['hll'])


I am working on the file "Shallow_water_approximate.ipynb".
The first task is:
In the next example, the left and right states are constructed so as to be connected exactly by a single right-going shock.  As we have discussed, the solver captures this solution exactly.
Can you write Python code for it?

# Isolated 2-shock example
h_l = 2.20698770767; h_r = 1.
u_l = 2.27057814896/h_l; hu_r = 0.

hu_l = h_l*u_l; hu_r = h_r*u_r
q_l = np.array([h_l, hu_l]); q_r = np.array([h_r, hu_r])

states = compare_solutions(q_l, q_r, ['Exact','HLL'])

sw.phase_plane_plot(q_l,q_r,g=1.,y_axis='u',
                    approx_states=states['hll'])


I am working on the file "Shallow_water_approximate.ipynb".
The first task is:
Notice that, by design, the HLLE wave always lies exactly on the leading edge of a rarefaction wave, when compared to the true solution.
The momentum solution above has the interesting feature that the solution in the 1-rarefaction is not monotone but appears as a "bump".  This can be explained by plotting the solution in the h-hu phase plane, where one sees that the integral curve has a local maximum between $q_\ell$ and $q_m$:
Can you write Python code for it?


I am working on the file "Shallow_water_approximate.ipynb".
The first task is:
The final example below illustrates that the HLLE solver maintains positivity when the same initial data is used as in the example with the Roe solver above where a negative intermediate depth was obtained.
Can you write Python code for it?

h_l = 1; h_r = 1
u_l = -1.5; u_r = 1.5

hu_l = h_l*u_l; hu_r = h_r*u_r
q_l = np.array([h_l, hu_l]); q_r = np.array([h_r, hu_r])

states = compare_solutions(q_l, q_r, ['Exact','HLL'])

sw.phase_plane_plot(q_l,q_r,g=1.,y_axis='u',
                    approx_states=states['hll'])


I am working on the file "Shallow_water_approximate.ipynb".
The first task is:
The final example includes a transonic rarefaction.
Can you write Python code for it?

# Transonic rarefaction example a
h_l = 1; h_r = 1
u_l = 0.5; u_r = 2

hu_l = h_l*u_l; hu_r = h_r*u_r
q_l = np.array([h_l, hu_l]); q_r = np.array([h_r, hu_r])

states = compare_solutions(q_l, q_r, ['Exact','HLL'])

sw.phase_plane_plot(q_l,q_r,g=1.,y_axis='u',approx_states=states['hll'])


I am working on the file "Shallow_water_approximate.ipynb".
The first task is:
The next example is a dam-break problem; the rarefaction wave is transonic because initially the water is all moving to the right.
Can you write Python code for it?

h_l = 4; h_r = 1
u_l = 1; u_r = 1

q_l, q_r, states, no_efix_states = compare_efix(h_l,u_l,h_r,u_r)

fig, ax = plt.subplots(1,2,figsize=(10,4))
sw.phase_plane_plot(q_l,q_r,g=1.,y_axis='u',
                    approx_states=no_efix_states,ax=ax[0])
sw.phase_plane_plot(q_l,q_r,g=1.,y_axis='u',
                    approx_states=states,ax=ax[1],color='orange')
ax[0].set_title('Without entropy fix')
ax[1].set_title('With entropy fix');


I am working on the file "Shanghai.ipynb".
The first task is:
Dix inversion
To relate measured and true velocities, consider reflection ray tracing in a layered velocity medium. Inside one layer with depth $\Delta z$ and velocity $v$, the time increment is
$$\Delta t = \displaystyle 2 \frac{\Delta z}{v\,\cos{\theta}}\;,$$
where $\theta$ is the ray angle.
The offset increment is
$$\Delta x = 2 \Delta z \tan{\theta}\;.$$
Between two layers with different velocities, the angle changes but satisfies the Snell's law
$$\displaystyle p = \frac{\sin{\theta_1}}{v_1} = \frac{\sin{\theta_2}}{v_2}$$
Expressing the angle using
$$\sin{\theta_n} = p\,v_n\;,$$
we can write parametric expressions for the reflection traveltime
$$\left{\begin{array}{rcl} t(p) & = & \displaystyle 2\,\sum_n \frac{\Delta z_n}{v_n\,\sqrt{1-p^2\,v_n^2}}\;, \
x(p) & = & \displaystyle 2\,\sum_n \frac{p\,v_n\,\Delta z_n}{\sqrt{1-p^2\,v_n^2}}\;.\end{array}\right.$$
Recal the power series expansion
$$(1+x)^q = \displaystyle 1 + q\,x + \frac{q\,(q-1)}{2}\,x^2 + \cdots$$
The expansion of the inverse square root in the equations above in powers of $p$ is
$$\displaystyle \frac{1}{\sqrt{1-p^2\,v_n^2}} = \left(1-p^2\,v_n^2\right)^{-1/2} = 1 + \frac{1}{2}\,p^2\,v_n^2 + \frac{3}{8}\,p^4\,v_n^4 + \cdots$$
Substituting in the parametric expressions, we can expand them in powers of $p$:
$$\begin{array}{rcl} t(p) & = & \displaystyle M_{-1} + \frac{1}{2}\,M_1\,p^2 + \frac{3}{8}\,M_3\,p^4 + \cdots \
x(p) & = & \displaystyle M_1\,p + \frac{1}{2}\,M_3\,p^3 + \frac{3}{8}\,M_5\,p^5 + \cdots\end{array}$$
where $M_k = \displaystyle 2\,\sum_n v_n^k \Delta z_n$.
We can invert the second series to express the ray parameter $p$ as a power series in powers of offset $x$:
$$p = a_1\,x + a_3\,x^3 + \cdots$$
Making a substitution and collecting powers of $x$:
$$x = \displaystyle M_1\,\left(a_1\,x + a_3\,x^3 + \cdots\right) + \frac{M_3}{2}\,\left(a_1\,x + a_3\,x^3 + \cdots\right)^3 + \cdots =  M_1\,a_1\,x + \left(M_1\,a_3 + \frac{M_3}{2}\,a_1^3\right)\,x^3 + \cdots$$
which leads to
$$\begin{array}{rcl} a_1 & = & \displaystyle \frac{1}{M_1} \...
Can you write Python code for it?

%%file vrms.scons

Flow('vrms','vel','mul $SOURCE | causint | math output="sqrt(input*0.004/(x1+0.004))" ')

Result('vrms','vel vrms vpick',
    '''
    cat axis=2 ${SOURCES[1:3]} |
    graph dash=0,0,1 title="RMS Velocity" label2=Velocity unit2=km/s
    transp=y yreverse=y wherexlabel=t wheretitle=b min2=0 max2=5
    ''')

view('vrms')

%%file dix.scons

# Find position of reflectors
Flow('refs','nmo','stack | envelope | max1 | window n1=4 | real | pad beg1=1')

# Find RMS velocities for layers
Flow('vn','vrms refs','inttest1 coord=${SOURCES[1]} interp=spline nw=4')

Plot('picks','refs vn',
     '''
     cmplx ${SOURCES[1]} |
     graph symbol=o symbolsz=8 wanttitle=n wantaxis=n plotcol=3
     transp=y yreverse=y min1=0 max1=2 min2=0 max2=5 pad=n
     ''')

# Find layer time thicknesses
Flow('dt','refs','igrad')

# Dix inversion
Flow('vdixn','vn refs dt',
     'mul ${SOURCES[:2]} | igrad | div ${SOURCES[2]} | math output="sqrt(input)" ')

Flow('vdix','vdixn refs',
     '''
     pad beg1=1 | igrad | window n1=4 |
     bin1 head=${SOURCES[1]} x0=0 dx=0.004 nx=501 | causint
     ''')

Plot('vdix','vel vrms vdix',
     '''
     cat axis=2 ${SOURCES[1:3]} |
     graph dash=0,0,1 title="Dix Inversion" label2=Velocity unit2=km/s
     transp=y yreverse=y wherexlabel=t wheretitle=b min2=0 max2=5 pad=n
     ''')

Result('vdix','vdix picks','Overlay')

from m8r import view
view('vdix')


I am working on the file "Shape_comparison.ipynb".
The first task is:
Heroin
Can you write Python code for it?

idx_most = similar_mols[0][0]
molecules[idx_most]


I am working on the file "Shape_comparison.ipynb".
The first task is:
Now we can use the usr function.
Can you write Python code for it?


I am working on the file "Shape_comparison.ipynb".
The first task is:
The least similar molecule
Similarity between these molecules:
Can you write Python code for it?

usr_most = usr(molecules[idx_most])
usr_least = usr(molecules[idx_least])
usr_similarity(usr_most, usr_least)


I am working on the file "Shape_comparison.ipynb".
The first task is:
The similarity function returns a number in range (0, 1], where a higher number means that the molecules are more similar and 1 means that the molecules have identical shapes.<br/>
All methods (USR, USR-CAT and Electroshape) use the same similarity function.
We will find a molecule similar to oxamide.
Can you write Python code for it?

similar_mols = []
for i, mol in enumerate(molecules):
    sim = usr_similarity(usr_heroin, usr(mol))
    similar_mols.append((i, sim))

similar_mols.sort(key=lambda similarity: similarity[1], reverse=True)
similar_mols

heroin


I am working on the file "Shape_comparison.ipynb".
The first task is:
To compute the shape using USR we need the molecule's 3D coordinates.
Can you write Python code for it?

heroin.make3D()
heroin.removeh()
for mol in molecules:
    mol.make3D()
    mol.removeh()