Reproducible model training: deep dive | by Vladislav Grozin | Towards Data Science

Reproducible research is easy. Just log your parameters and metrics somewhere, fix seeds, and you are good to go
— me, about two weeks ago.
Oh boy, I was wrong.
There are a lot of workshops, tutorials, and conferences on reproducible research.
A plethora of utilities, tools, and frameworks are made to help us make nice reproducible solutions.
However, there are still problems. These pitfalls are not apparent in a simple tutorial project but bound to happen in any real research. Very few people talk about them, so I’d like to share my knowledge about these topics.
In this post, I’ll tell a story about my quest towards being able to consistently train models (so it gives the same weights every run).
Once upon a time, I had a project related to computer vision (handwriting author identification).
At some point, I decided to spend time refactoring code and tidying up the project. I split my large Keras model into several stages, designed test sets for each, and used ML Flow to track results of and performance of each stage (it was quite hard — but that’s a story for another day).
After a week or so of refactoring, I have built a nice pipeline, caught a few bugs, managed to fiddle a bit with hyperparameters and slightly improved performance.
However, I noticed one strange thing. I fixed all random seeds, as numerous guides suggested:
def fix_seeds(seed):    random.seed(seed)    np.random.seed(seed)    tf.set_random_seed(seed)    session_conf = tf.ConfigProto(intra_op_parallelism_threads=1, inter_op_parallelism_threads=1)    sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)    K.set_session(sess)
But for some reason, two consecutive runs with identical hyperparameters gave different results.
Being unable to track the issue down in the project, I decided to make a script with a small model that reproduces the issue.
I defined a simple neural network:
def create_mlp(dim):    model = Sequential()    model.add(Dense(8, input_dim=dim))    model.add(Dense(1))    return model
Because data does not matter much here, I generated some random data to work with. After that we are ready to train our model:
model = create_mlp(10)init_weights = np.array(model.get_weights()[0]).sum()model.compile(optimizer=keras.optimizers.RMSprop(lr=1e-2),              loss=keras.losses.MSE)model.fit(Xs, Ys, batch_size=10, epochs=10)
After training, we can check reproducibility. assert_same_across_runs is a simple function that checks whether the passed value is the same between runs (it does so by writing the value to a file):
assert_same_across_runs("dense model data", Ys.sum())assert_same_across_runs("dense model weight after training",     init_weights)assert_same_across_runs("dense model weight after training",     np.array(model.get_weights()[0]).sum())
Full script
I ran it several times. Model weights were exactly the same each execution.
Strange! I added a bit of complexity into the model by plugging in convolution layers:
def create_nnet(dim):    input = Input(shape=dim)    conv = Conv2D(5, (3, 3), activation="relu")(input)    flat = Flatten()(conv)    output = Dense(1)(flat)    return Model([input], [output])
Full script
Training procedure quite is similar: we create the model, generate some data, train our model, and then check metrics.
And, voila, it broke. Every time the script is run, a different number is printed.
This issue does not happen to machines without GPU. If your machine has GPU, you can hide it from the script by setting CUDA_VISIBLE_DEVICES environment variable to “” from the console.
A quick investigation discovered ugly truth: there are issues with reproducibility, and some layers make model irreproducible, at least by default(to all “you suffer because you use Keras”, Pytorch has a similar problem)
Some complex operations do not have a well-defined order of sub-operations.For example, convolution is just a bunch of additions, but the order of these additions is not defined.
So, each execution results in different order of summations. Because we operate with floating-point with finite precision, convolution yields slightly different results.
Yep, order of summation matters, (a+b)+c != a+(b+c)! You can even check it yourself:
import numpy as npnp.random.seed(42)xs = np.random.normal(size=10000)a = 0.0for x in xs:    a += xb = 0for x in reversed(xs):    b += xprint(a, b)print("Difference: ", a - b)
Should print
-21.359833684261957 -21.359833684262377Difference: 4.192202140984591e-13
Yes, it is “just 4e-13”. But because this imprecision happens at each level of the deep neural network, and for each batch, this error accumulates over layers and time, and model weights diverge significantly. As a result, losses of two consecutive runs can look like this:
One might argue that these small discrepancies between runs should not affect the model performance, and in fact, fixing random seed and exact reproducibility is not that important.
Well, there are merits to this argument. Randomness affects weights; so, model performance technically depends on the random seed. Changing the seed should have a lesser impact on accuracy than adding new featuresor changing architecture. Moreover, because the random seed is not an essential part of the model, it might be useful to evaluate model several times for different seeds (or let GPU randomize), and report averaged values along with confidence intervals.
However, in practice, very few papers do that. Instead, models are compared with a baseline based on point estimates. Moreover, there are concerns that improvements that papers report are less than this randomness, and improper aggregation can skew results.
What’s even worse is that results in irreducible stochasticity in the code. You cannot write unit tests. It usually does not bother data scientists much, so instead imagine a situation.
You have found a paper that describes a fancy model — and it has cleanly organized open-source implementation. You download the code and the model, start training it. After a couple of days (it’s a very fancy model)you test the model. It does not work. Is it a problem with the hyperparameters? With your hardware or driver versions? Dataset? Maybe, repo authors are frauds, and the problem is in the repository itself? You’ll never know.
Until experiments are completely reproducible, troubleshooting is extremely cumbersome, as you do not know which part of the pipeline contains the problem.
So, it seems that having the ability to reproduce model can be extremely handy.
Order of summations is not defined? Okay. We can define it ourselves, i.e. rewrite convolution as a bunch of summations. Yes, it comes with some overhead, but it would solve our problem.
Happily, CuDNN already has “reproducible” implementation of most operations (and it is, indeed, slower). First of all, you do not have to write anything yourself, you just have to tell CuDNN to use it. Secondly, CuDNN ispretty low in the stack — so, you won’t get much overhead:
There is a layered pipe between your code and your hardware
Because we do not interact with CuDNN directly, we have to tell our library of choice to use specific implementations. In other words, we have to turn on the “I’m ok with slower training, give me consistent results every run” flag.
Unfortunately, Keras does not have that functionality (yet), as described in these issues.
Looks like it’s time for PyTorch to shine. It has settings that will enable the use of CuDNN deterministic implementations:
Let’s write a simple network with a single convolution, and train it on random data. The exact architecture or data do not matter much, as we are just testing reproducibility.
class Net(nn.Module):    def __init__(self, in_shape: int):        super(Net, self).__init__()        self.conv1 = nn.Conv2d(3, 5, 3)        self.hidden_size = int((in_shape — 2) * (in_shape — 2) / 4) * 5        self.fc1 = nn.Linear(self.hidden_size, 1)def forward(self, x):        x = F.relu(self.conv1(x))        x = F.max_pool2d(x, 2)        x = x.view(-1, self.hidden_size)        x = F.relu(self.fc1(x))        return x
the fix_seeds function also gets changed to include
def fix_seeds(seed):    random.seed(seed)    np.random.seed(seed)    torch.manual_seed(42)    torch.backends.cudnn.deterministic = True    torch.backends.cudnn.benchmark = False
Again, we’ll use synthetic data to train the network.
After initialization, we ensure that the sum of weights is equal to a specific value. Similarly, after training the network we check the model weights. If there is any discrepancies or randomness, the script will tell us.
Full script
python3 reproducibility_cnn_torch.py
gives consistent results, which means training neural network in PyTorch gives exactly the same weights.
There are caveats. First of all, these settings can impact training time, but tests show a negligible difference (1.14 +- 0.07 seconds for non-deterministic version; 1.17 +- 0.08 seconds for deterministic one). Secondly, CuDNN documentation warns us that there are several algorithms without reproducibility guarantees. These algorithms are usually faster than their deterministic variations, but PyTorch does not use them if flags are set. At last, some modules are non-deterministic (I was unable to reproduce the issue myself).
Anyways, these issues are rarely an obstruction. So, Torch has built-in reproducibility support, and Keras does not have such functionality right now.
But wait! There are some speculations that because Keras and Torch share back-end, setting these variables in PyTorch will affect Keras. It could be the case if the variables affected some process-wide parameters. This way, we can make use of these flags AND use Keras to write models painlessly.First of all, let’s check out how these flags work.
Let’s go deeper! Setting deterministic / benchmark flags from Python sidecalls functions defined in the C sources. Here, the state is stored internally.After that, this flag affects the choice of the algorithm during convolution. In other words, setting this flag does not affect the global state (environment variables, for example).
So, it seems that PyTorch settings do not affect Keras internals (and actual tests confirm that).
Added: some users report that for some layers (CuDNNLSTM is a notable example) and for some hardware and software combinations this hack MAY work. This is definitely an interesting effect, but I would discourage using this in practice. Using a technique that sometimes works and sometimes doesn’t undermines the whole idea of reproducible learning.
Fixing random seeds is extremely useful, as it helps with model debugging. Moreover, the decreased metric variance between runs helps to make decisions about whether some feature increases the performance of the model.
But sometimes it is not enough. Some models (convolutional, recurrent) require additional CuDNN settings to be set. Keras has no such functionality yet, while PyTorch does:
torch.backends.cudnn.deterministic = Truetorch.backends.cudnn.benchmark = False