RuntimeError: one of the variables needed for ensemble neural net gradient computation has been modified by an inplace operation

Mlohens257: <p>I am trying to train an ensemble of neural networks in PyTorch. During the backward pass, I am encountering the following error message:<br>
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [240, 1]], which is output 0 of AsStridedBackward0, is at version 2; expected version 1 instead.</p>
<p>Here is the code that is causing the error:</p>
<p>import torch<br>
import torch.nn as nn<br>
import torch.optim as optim<br>
from torch.utils.tensorboard import SummaryWriter<br>
from sklearn import datasets<br>
from sklearn.model_selection import train_test_split<br>
from sklearn.preprocessing import StandardScaler<br>
from sklearn.utils import shuffle<br>
import numpy as np</p>
<h1>
<a name="check-if-a-gpu-is-available-1" class="anchor" href="#check-if-a-gpu-is-available-1"></a>Check if a GPU is available</h1>
<p>device = torch.device(“cuda” if torch.cuda.is_available() else “cpu”)<br>
print(“Using device:”, device)</p>
<h1>
<a name="load-the-dataset-possible-values-iris-digits-wine-breast_cancer-diabetes-boston-2" class="anchor" href="#load-the-dataset-possible-values-iris-digits-wine-breast_cancer-diabetes-boston-2"></a>Load the dataset (Possible values: ‘iris’, ‘digits’, ‘wine’, ‘breast_cancer’, ‘diabetes’, ‘boston’)</h1>
<p>dataset_name = ‘breast_cancer’<br>
X, y = load_dataset(dataset_name)</p>
<h1>
<a name="shuffle-and-split-the-dataset-into-training-and-testing-sets-3" class="anchor" href="#shuffle-and-split-the-dataset-into-training-and-testing-sets-3"></a>Shuffle and split the dataset into training and testing sets</h1>
<p>X, y = shuffle(X, y, random_state=42)<br>
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)</p>
<h1>
<a name="standardize-the-data-zero-mean-unit-variance-4" class="anchor" href="#standardize-the-data-zero-mean-unit-variance-4"></a>Standardize the data (zero mean, unit variance)</h1>
<p>scaler = StandardScaler()<br>
X_train = scaler.fit_transform(X_train)<br>
X_test = scaler.transform(X_test)</p>
<h1>
<a name="convert-numpy-arrays-to-pytorch-tensors-5" class="anchor" href="#convert-numpy-arrays-to-pytorch-tensors-5"></a>Convert NumPy arrays to PyTorch tensors</h1>
<p>X_train = torch.tensor(X_train, dtype=torch.float32).to(device)<br>
y_train = torch.tensor(y_train, dtype=torch.long).to(device)<br>
X_test = torch.tensor(X_test, dtype=torch.float32).to(device)<br>
y_test = torch.tensor(y_test, dtype=torch.long).to(device)</p>
<h1>
<a name="define-a-simple-neural-network-6" class="anchor" href="#define-a-simple-neural-network-6"></a>Define a simple neural network</h1>
<p>class SimpleNN(nn.Module):<br>
def <strong>init</strong>(self, input_size, hidden_size, output_size):<br>
super(SimpleNN, self).<strong>init</strong>()<br>
self.fc1 = nn.Linear(input_size, hidden_size)<br>
self.relu1 = nn.ReLU()<br>
self.fc2 = nn.Linear(hidden_size, hidden_size)<br>
self.relu2 = nn.ReLU()<br>
self.fc3 = nn.Linear(hidden_size, hidden_size)<br>
self.relu3 = nn.ReLU()<br>
self.fc4 = nn.Linear(hidden_size, hidden_size)<br>
self.relu4 = nn.ReLU()<br>
self.fc5 = nn.Linear(hidden_size, hidden_size)<br>
self.relu5 = nn.ReLU()<br>
self.fc6 = nn.Linear(hidden_size, hidden_size)<br>
self.relu6 = nn.ReLU()<br>
self.fc7 = nn.Linear(hidden_size, hidden_size)<br>
self.relu7 = nn.ReLU()<br>
self.fc8 = nn.Linear(hidden_size, output_size)</p>
<pre><code>def forward(self, x):
    x = self.fc1(x)
    x = self.relu1(x)
    x = self.fc2(x)
    x = self.relu2(x)
    x = self.fc3(x)
    x = self.relu3(x)
    x = self.fc4(x)
    x = self.relu4(x)
    x = self.fc5(x)
    x = self.relu5(x)
    x = self.fc6(x)
    x = self.relu6(x)
    x = self.fc7(x)
    x = self.relu7(x)
    x = self.fc8(x)
    return x
</code></pre>
<h1>
<a name="create-a-list-of-neural-network-instances-7" class="anchor" href="#create-a-list-of-neural-network-instances-7"></a>Create a list of neural network instances</h1>
<p>input_size = X.shape[1]<br>
hidden_size = input_size*8<br>
output_size = 1<br>
ensemble_size = 20<br>
ensemble = [SimpleNN(input_size, hidden_size, output_size).to(device) for i in range(ensemble_size)]</p>
<h1>
<a name="enable-anomaly-detection-8" class="anchor" href="#enable-anomaly-detection-8"></a>Enable anomaly detection</h1>
<p>torch.autograd.detect_anomaly()</p>
<h1>
<a name="specify-number-of-epochs-9" class="anchor" href="#specify-number-of-epochs-9"></a>Specify number of epochs</h1>
<p>epochs = 7500</p>
<p>def entropy(p):<br>
eps = 1e-9<br>
return -p * torch.log2(p + eps) - (1 - p) * torch.log2(1 - p + eps)</p>
<p>def step_sigmoid(x):<br>
return torch.sigmoid(1000*(x -.5))</p>
<p>def voted_predict(y_pred):<br>
return step_sigmoid(torch.mean(step_sigmoid(y_pred), axis = 1))</p>
<h1>
<a name="compute-the-indicator-function-i-10" class="anchor" href="#compute-the-indicator-function-i-10"></a>Compute the indicator function I</h1>
<p>def indicator(y_pred, y_true):<br>
return (voted_predict(y_pred)- y_true)**2</p>
<h1>
<a name="calculate-the-probability-11" class="anchor" href="#calculate-the-probability-11"></a>Calculate the probability</h1>
<p>def positive_probability(y_pred):<br>
num_pos = torch.sum(step_sigmoid(y_pred), axis = 1)<br>
total = y_pred.shape[1]<br>
p_pos = num_pos / total<br>
return p_pos</p>
<p>def concatenate_predictions(ens_y_pred, y_pred):<br>
if ens_y_pred is None:<br>
ens_y_pred = y_pred.clone()<br>
else:<br>
# Create a new tensor to hold the concatenated values<br>
new_ens_y_pred = torch.cat([ens_y_pred, y_pred], dim=1)</p>
<pre><code>    # Return the new tensor
    ens_y_pred = new_ens_y_pred
return ens_y_pred
</code></pre>
<h1>
<a name="define-the-custom-loss-function-12" class="anchor" href="#define-the-custom-loss-function-12"></a>Define the custom loss function</h1>
<p>def custom_loss(y_true, y_pred):</p>
<pre><code># Make a copy of y_pred before passing it to the indicator function
y_pred_copy = y_pred#.clone().detach()
y_true_copy = y_true#.clone().detach()  

I = indicator(y_pred_copy, y_true_copy).detach().clone()
print("I in custom_loss:", I)

# Calculate the probability
p_pos = positive_probability(y_pred_copy)
print("p_pos in custom_loss:", p_pos)

# Compute the entropy of the predicted probabilities
H = entropy(p_pos)#.detach().clone()
print("H in custom_loss:", H)

# Compute the loss function
loss = I + torch.pow(H, 1 - I) * torch.pow(1 - H, I)
print("loss in custom_loss:", loss)
print("torch.mean(loss):", torch.mean(loss))

return torch.mean(loss)         
</code></pre>
<p>optimizer = optim.SGD([{‘params’: model.parameters()} for model in ensemble], lr=0.01)</p>
<h1>
<a name="initialize-tensorboard-writer-13" class="anchor" href="#initialize-tensorboard-writer-13"></a># Initialize TensorBoard writer</h1>
<h1>
<a name="writer-summarywriter-14" class="anchor" href="#writer-summarywriter-14"></a>writer = SummaryWriter()</h1>
<h1>
<a name="train-the-ensemble-of-neural-networks-15" class="anchor" href="#train-the-ensemble-of-neural-networks-15"></a># Train the ensemble of neural networks</h1>
<h1>
<a name="prev_weights-16" class="anchor" href="#prev_weights-16"></a>prev_weights = {}</h1>
<p>for epoch in range(epochs):<br>
if epoch%100 == 0:<br>
print(“epoch:”, epoch)</p>
<h1>
<a name="shuffle-the-training-data-for-each-epoch-17" class="anchor" href="#shuffle-the-training-data-for-each-epoch-17"></a># Shuffle the training data for each epoch</h1>
<h1>
<a name="x_train_shuffled-y_train_shuffled-shufflex_trainclone-y_trainclone-random_stateepoch-18" class="anchor" href="#x_train_shuffled-y_train_shuffled-shufflex_trainclone-y_trainclone-random_stateepoch-18"></a>X_train_shuffled, y_train_shuffled = shuffle(X_train.clone(), y_train.clone(), random_state=epoch)</h1>
<h1>
<a name="initialize-the-ensemble-predictions-19" class="anchor" href="#initialize-the-ensemble-predictions-19"></a># Initialize the ensemble predictions</h1>
<h1>
<a name="ens_y_pred-none-20" class="anchor" href="#ens_y_pred-none-20"></a>ens_y_pred = None</h1>
<h1>
<a name="train-each-neural-network-in-the-ensemble-21" class="anchor" href="#train-each-neural-network-in-the-ensemble-21"></a>Train each neural network in the ensemble</h1>
<p>for i, model in enumerate(ensemble):<br>
# Set model to training mode<br>
model.train()</p>
<pre><code>  # Forward pass
  y_pred = model(X_train)
  print("y_pred.shape:", y_pred.shape)

  # Concatenate the tensors along the first dimension
  ens_y_pred = concatenate_predictions(ens_y_pred, y_pred)

  # Reshape tensor
</code></pre>
<h1>
<a name="ens_y_pred-ens_y_predreshapeens_y_predshape1ens_y_predshape0-22" class="anchor" href="#ens_y_pred-ens_y_predreshapeens_y_predshape1ens_y_predshape0-22"></a>ens_y_pred = ens_y_pred.reshape([ens_y_pred.shape[1],ens_y_pred.shape[0]])</h1>
<p>print(“ens_y_pred.shape:”,ens_y_pred.shape )</p>
<p>for i, model in enumerate(ensemble):<br>
print(‘i:’, i)<br>
# Calculate loss</p>
<pre><code>  loss = custom_loss(y_train, ens_y_pred)

  # Backward pass and optimization
  optimizer.zero_grad()
  loss.backward(retain_graph=True)
  optimizer.step()
</code></pre>

ptrblck: <p>Often using <code>retain_graph=True</code> is causing these issues as e.g. the parameters were already updated and stale forward activations are used.<br>
Could you explain why you are using this argument?</p>

Mlohens257: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, Thank you for your prompt response. If I don’t use retain_graph = True, then I get the following error:</p>
<p>RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.</p>

ptrblck: <p>This seems to be the actual issue then.<br>
Your code is unfortunately not formatted properly so hard to read (you can post code snippets by wrapping them into three backticks ```), but generally these issues are raised if you are reusing tensors attached to a computation graph and are then calling <code>backward</code> multiple times.</p>

Mlohens257: <p>Ah, thank you for letting me know. I’ve included the formatted code below. Does this have anything to do with the fact that I’m sharing a tensor loss for all the models in my ensemble.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.tensorboard import SummaryWriter
from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.utils import shuffle
import numpy as np

# Check if a GPU is available
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print("Using device:", device)

# Load the dataset (Possible values: 'iris', 'digits', 'wine', 'breast_cancer', 'diabetes', 'boston')
dataset_name = 'breast_cancer'
X, y = load_dataset(dataset_name)

# Print data shapes
print("Dataset Name:", dataset_name)
print("Input Shape:", X.shape)
print("Possible Outputs:", len(np.unique(y)))
print("Unique Outputs:", np.unique(y))
print(" ")

# Shuffle and split the dataset into training and testing sets
X, y = shuffle(X, y, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Standardize the data (zero mean, unit variance)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

# Convert NumPy arrays to PyTorch tensors
X_train = torch.tensor(X_train, dtype=torch.float32).to(device)
y_train = torch.tensor(y_train, dtype=torch.long).to(device)
X_test = torch.tensor(X_test, dtype=torch.float32).to(device)
y_test = torch.tensor(y_test, dtype=torch.long).to(device)

# Define a simple neural network
class SimpleNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(SimpleNN, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.relu1 = nn.ReLU()
        self.fc2 = nn.Linear(hidden_size, hidden_size)
        self.relu2 = nn.ReLU()
        self.fc3 = nn.Linear(hidden_size, hidden_size)
        self.relu3 = nn.ReLU()
        self.fc4 = nn.Linear(hidden_size, hidden_size)
        self.relu4 = nn.ReLU()
        self.fc5 = nn.Linear(hidden_size, hidden_size)
        self.relu5 = nn.ReLU()
        self.fc6 = nn.Linear(hidden_size, hidden_size)
        self.relu6 = nn.ReLU()
        self.fc7 = nn.Linear(hidden_size, hidden_size)
        self.relu7 = nn.ReLU()
        self.fc8 = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        x = self.fc1(x)
        x = self.relu1(x)
        x = self.fc2(x)
        x = self.relu2(x)
        x = self.fc3(x)
        x = self.relu3(x)
        x = self.fc4(x)
        x = self.relu4(x)
        x = self.fc5(x)
        x = self.relu5(x)
        x = self.fc6(x)
        x = self.relu6(x)
        x = self.fc7(x)
        x = self.relu7(x)
        x = self.fc8(x)
        return x

# Create a list of neural network instances
input_size = X.shape[1]
hidden_size = input_size*8
output_size = 1
ensemble_size = 20
ensemble = [SimpleNN(input_size, hidden_size, output_size).to(device) for i in range(ensemble_size)]

# Enable anomaly detection
torch.autograd.detect_anomaly(True)

# Specify number of epochs
epochs = 7500

def entropy(p):
    eps = 1e-9
    return -p * torch.log2(p + eps) - (1 - p) * torch.log2(1 - p + eps)

def step_sigmoid(x):
  return torch.sigmoid(1000*(x -.5))

def voted_predict(y_pred):
  return step_sigmoid(torch.mean(step_sigmoid(y_pred), axis = 1))

# Compute the indicator function I
def indicator(y_pred, y_true):
    return (voted_predict(y_pred)- y_true)**2

# Calculate the probability
def positive_probability(y_pred):
  num_pos = torch.sum(step_sigmoid(y_pred), axis = 1)
  total = y_pred.shape[1]
  p_pos = num_pos / total
  return p_pos

def concatenate_predictions(ens_y_pred, y_pred):
    if ens_y_pred is None:
        ens_y_pred = y_pred.clone()
    else:
        # Create a new tensor to hold the concatenated values
        new_ens_y_pred = torch.cat([ens_y_pred, y_pred], dim=1)

        # Return the new tensor
        ens_y_pred = new_ens_y_pred
    return ens_y_pred
 

# Define the custom loss function
def custom_loss(y_true, y_pred):
    # print("y_true.shape in custom_loss:", y_true.shape)  
    # print("y_pred.shape in custom_loss:", y_pred.shape)

    # Make a copy of y_pred before passing it to the indicator function
    y_pred_copy = y_pred#.clone().detach()
    y_true_copy = y_true#.clone().detach()

    
    
    I = indicator(y_pred_copy, y_true_copy).detach().clone()
    # print("I in custom_loss:", I)

    # Calculate the probability
    p_pos = positive_probability(y_pred_copy)
    # print("p_pos in custom_loss:", p_pos)

    # Compute the entropy of the predicted probabilities
    H = entropy(p_pos)#.detach().clone()
    # print("H in custom_loss:", H)

    # Compute the loss function
    loss = I + torch.pow(H, 1 - I) * torch.pow(1 - H, I)
    print("loss in custom_loss:", loss)
    # print("torch.mean(loss):", torch.mean(loss))

    return torch.mean(loss)         

#optimizer = optim.SGD([{'params': model.parameters()} for model in ensemble], lr=0.01)
optimizer = optim.SGD([{'params': model.parameters()} for model in ensemble], lr=0.01)
# optimizer = torch.optim.SGD(model.parameters(), lr=1e-5)
# or
# optimizer = torch.optim.SGD(model.classifier.parameters(), lr=1e-5)

# # Initialize TensorBoard writer
# writer = SummaryWriter()


# # Train the ensemble of neural networks
# prev_weights = {}

for epoch in range(epochs):
  if epoch%100 == 0:
    print("epoch:", epoch)



  # Shuffle the training data for each epoch
  # X_train_shuffled, y_train_shuffled = shuffle(X_train.clone(), y_train.clone(), random_state=epoch)
  # X_train = X_train_shuffled.clone()
  # y_train = y_train_shuffled.clone()
  # Initialize the ensemble predictions
  ens_y_pred = None

  # Train each neural network in the ensemble
  for i, model in enumerate(ensemble):
      # Set model to training mode
      model.train()

      # Forward pass
      y_pred = model(X_train)
      # print("y_pred.shape:", y_pred.shape)

      # Concatenate the tensors along the first dimension
      ens_y_pred = concatenate_predictions(ens_y_pred, y_pred)

      # Reshape tensor
  # ens_y_pred = ens_y_pred.reshape([ens_y_pred.shape[1],ens_y_pred.shape[0]])

  print("ens_y_pred.shape:",ens_y_pred.shape )

  for i, model in enumerate(ensemble):      
      print('i:', i)
      # Calculate loss

      loss = custom_loss(y_train, ens_y_pred)

      # Backward pass and optimization
      optimizer.zero_grad()
      # loss.backward(retain_graph=True)
      loss.backward()
      optimizer.step()
      

      # # Record the training loss for this model
      # writer.add_scalar(f'Training Loss (Model {i + 1})', loss.item(), epoch)

      # # Record the requested information for this model
      # for j, layer in enumerate(model.children()):
      #     if isinstance(layer, nn.Linear):
      #         # Change in weights
      #         if (i, j) in prev_weights:
      #             weight_diff = layer.weight.detach() - prev_weights[(i, j)]['weight']
      #             writer.add_histogram(f'Model {i + 1}/Layer {j + 1}/Weight Change', weight_diff, epoch)
      #         prev_weights[(i, j)] = {'weight': layer.weight.detach().clone(), 'bias': layer.bias.detach().clone()}

      #         # Store weights
      #         if (i, j) in prev_weights:
      #             weight_ = layer.weight.detach()
      #             writer.add_histogram(f'Model {i + 1}/Layer {j + 1}/Weights', weight_, epoch)

      #         # Variance between input and output
      #         input_var = np.var(layer_outputs[i][j - 1]) if j &gt; 0 else np.var(X_train.cpu().numpy())
      #         output_var = np.var(layer_outputs[i][j])
      #         writer.add_scalars(f'Model {i + 1}/Layer {j + 1}/Variance', {'input': input_var, 'output': output_var}, epoch)

      #         # Neuron standard deviation
      #         neuron_std = np.std(layer_outputs[i][j], axis=0)
      #         writer.add_histogram(f'Model {i + 1}/Layer {j + 1}/STD', torch.tensor(neuron_std), epoch)

      #         # Bias of each layer
      #         writer.add_histogram(f'Model {i + 1}/Layer {j + 1}/Bias', layer.bias.detach(), epoch)

              # # Entropy for each layer
              # if j == len(model) - 1:  # Only apply softmax to the output layer
              #     softmax = nn.Softmax(dim=1)
              #     probs = softmax(y_pred).detach().cpu().numpy()
              # else:
              #     probs = layer_outputs[i][j]
            
              # layer_entropy = entropy(probs)
              # writer.add_scalar(f'Model {i + 1}/Layer {j + 1}/Entropy', layer_entropy, epoch)

# Evaluate the ensemble of neural networks on the test set
with torch.no_grad():
    outputs = []
    for model in ensemble:
        model.eval()
        outputs.append(model(X_test))
    outputs = torch.stack(outputs)
    ensemble_output = torch.mean(outputs, dim=0)
    ensemble_loss = nn.functional.cross_entropy(ensemble_output, y_test)
    # writer.add_scalar('Ensemble Test Loss', ensemble_loss.item(), epoch)

    _, predicted = torch.max(ensemble_output, 1)
    correct = (predicted == y_test).sum().item()
    accuracy = correct / y_test.size(0)
    # writer.add_scalar('Ensemble Test Accuracy', accuracy, epoch)

</code></pre>

Mlohens257: <p>I think that the error (“Trying to backward through the graph a second time…”) may have been because of the torch.cat function that is was using previously. I updated my code by using a list to store the ensemble predictions, instead of the torch.cat method, as shown below. The code is running now, but the loss stays the same. Any suggestions <a class="mention" href="/u/ptrblck">@ptrblck</a>?</p>
<p>‘’’</p>
<h1>
<a name="initialize-the-ensemble-predictions-1" class="anchor" href="#initialize-the-ensemble-predictions-1"></a>Initialize the ensemble predictions</h1>
<h1>
<a name="ens_y_pred-none-2" class="anchor" href="#ens_y_pred-none-2"></a>ens_y_pred = None</h1>
<p>ens_y_pred_list = []</p>
<h1>
<a name="train-each-neural-network-in-the-ensemble-3" class="anchor" href="#train-each-neural-network-in-the-ensemble-3"></a>Train each neural network in the ensemble</h1>
<p>for i, model in enumerate(ensemble):<br>
# Set model to training mode<br>
model.train()</p>
<pre><code>  # Forward pass
  y_pred = model(X_train)
  # print("y_pred.shape:", y_pred.shape)

  # Concatenate the tensors along the first dimension
  # ens_y_pred = concatenate_predictions(ens_y_pred, y_pred)
  ens_y_pred_list.append(y_pred)

  # Reshape tensor
</code></pre>
<h1>
<a name="ens_y_pred-ens_y_predreshapeens_y_predshape1ens_y_predshape0-4" class="anchor" href="#ens_y_pred-ens_y_predreshapeens_y_predshape1ens_y_predshape0-4"></a>ens_y_pred = ens_y_pred.reshape([ens_y_pred.shape[1],ens_y_pred.shape[0]])</h1>
<h1>
<a name="printens_y_predshapeens_y_predshape-5" class="anchor" href="#printens_y_predshapeens_y_predshape-5"></a>print(“ens_y_pred.shape:”,ens_y_pred.shape )</h1>
<p>for i, model in enumerate(ensemble):</p>
<pre><code>  # print('i:', i)
  # Calculate loss
  y_pred = ens_y_pred_list[i]
  loss = custom_loss(y_train, y_pred)
  # loss = custom_loss(y_train, ens_y_pred)

  # Backward pass and optimization
  optimizer.zero_grad()
  # loss.backward(retain_graph=True)
  loss.backward()
  optimizer.step()
</code></pre>
<p>‘’’</p>

ptrblck: <p>The issue in the previously posted code is raised because you are creating <code>ens_y_pred</code> by concatenating the outputs from different models and try to call <code>backward</code> on it multiple times instead of once. The new code seems to treat each prediction individually and you could create a single <code>for</code> loop instead of storing the outputs on a <code>list</code> and iterate it afterwards.</p>
<p>Your model training is stuck as the intermediate losses are zero matrices when I execute the code, e.g. <code>step_sigmoid</code> is saturating the output to large negative values thus creating an output full of zeros, which then creates zeros in <code>H</code> etc.</p>