Woops I had made a small error in the model, where I only updated if the outlier_toggle was True. My bad!