Repository: keras-team/keras

Issue:
Unable to change validation dataset for "model.fit()" function after it raises an exception
Cross reporting issue #[62014](https://github.com/tensorflow/tensorflow/issues/62104) from tensorflow repo.

When using model.fit() with validation_data, during first call to fit() it will create an `EpochIterator` object for evaluation and cache it. This cache will be deleted after completion of all epochs.

However if we try to change the `validation_data` shapes (Not sure of the exact use case of this) and called `model.fit() `again it will raise an exception like Graph execution error which is intended. After this if we call `model.fit()` with correct `validation_data` that of first training call it won't work. The reason being is after first call is success evaluation `EpochIterator` will be deleted at the end and during second training call a new `EpochIterator` will be generated again with changed shape.But during `evaluate()` call it will raise an exception terminating the process without deleting evaluation `EpochIterator` object and it remains in cache. During third call though with correct `validation_data` it will not create new `EpochIterator` object as one already exists in cache which makes it fail again.

It don't strike me any use case of this situation. Reporting here whether it needs attention.

Attaching [gist](https://colab.sandbox.google.com/gist/SuryanarayanaY/767f89dd0b195367be917add09afc8d4/62104_with_keras-core_r2.ipynb) replicating the issue with `keras_core`.


Discussion:
If worth to consider this as bug, a simple fix may be initializing `EpochIterator` to `None` in `model.fit` at beginning itself.

In the source code self._eval_epoch_iterator is the validation_data EpochIterator object. Setting 
`self._eval_epoch_iterator = None` at the beginning of model.fit() ensures creation of new EpochIterator object for each training call. 

It indeeds fix this issue but not sure it has any toll on other things.
Thanks for this report. Can you create a PR for this issue?

Resolving patch:
diff --git a/keras/backend/jax/trainer.py b/keras/backend/jax/trainer.py
--- a/keras/backend/jax/trainer.py
+++ b/keras/backend/jax/trainer.py
@@ -352,6 +352,7 @@ def fit(
     ):
         self._assert_compile_called("fit")
         # TODO: respect compiled trainable state
+        self._eval_epoch_iterator = None
         if validation_split and validation_data is None:
             # Create the validation data using the training data. Only supported
             # for TF/numpy/jax arrays.
diff --git a/keras/backend/tensorflow/trainer.py b/keras/backend/tensorflow/trainer.py
--- a/keras/backend/tensorflow/trainer.py
+++ b/keras/backend/tensorflow/trainer.py
@@ -265,6 +265,7 @@ def fit(
     ):
         self._assert_compile_called("fit")
         # TODO: respect compiled trainable state
+        self._eval_epoch_iterator = None
         if validation_split and validation_data is None:
             # Create the validation data using the training data. Only supported
             # for TF/numpy/jax arrays.
diff --git a/keras/backend/torch/trainer.py b/keras/backend/torch/trainer.py
--- a/keras/backend/torch/trainer.py
+++ b/keras/backend/torch/trainer.py
@@ -221,6 +221,7 @@ def fit(
             )
 
         # TODO: respect compiled trainable state
+        self._eval_epoch_iterator = None
         if validation_split and validation_data is None:
             # Create the validation data using the training data. Only supported
             # for TF/numpy/jax arrays.


Test patch:
diff --git a/keras/trainers/trainer_test.py b/keras/trainers/trainer_test.py
--- a/keras/trainers/trainer_test.py
+++ b/keras/trainers/trainer_test.py
@@ -650,6 +650,49 @@ def call(self, inputs):
         out = model.predict({"a": x1, "b": x2})
         self.assertEqual(out.shape, (3, 4))
 
+    @pytest.mark.requires_trainable_backend
+    def test_for_eval_epoch_iterator(self):
+        model = ExampleModel(units=3)
+        model.compile(
+            optimizer="adam", loss="mse", metrics=["mean_absolute_error"]
+        )
+        x = np.ones((16, 4))
+        y = np.zeros((16, 3))
+        x_test = np.ones((16, 4))
+        y_test = np.zeros((16, 3))
+        model.fit(
+            x,
+            y,
+            batch_size=4,
+            validation_data=(x_test, y_test),
+        )
+        assert getattr(model, "_eval_epoch_iterator", None) is None
+
+        # Try model.fit with reshaped validation_data
+        # This will throw an exception which is intended
+        try:
+            model.fit(
+                x,
+                y,
+                batch_size=4,
+                validation_data=(
+                    x_test.reshape((-1, 16, 4)),
+                    y_test.reshape((-1, 16, 3)),
+                ),
+            )
+        except:
+            pass
+
+        # Try model.fit with correct validation_data this should work.
+        # After successful training `_eval_epoch_iterator` should be None
+        model.fit(
+            x,
+            y,
+            batch_size=4,
+            validation_data=(x_test, y_test),
+        )
+        assert getattr(model, "_eval_epoch_iterator", None) is None
+
     @pytest.mark.requires_trainable_backend
     def test_callback_methods_keys(self):
         class CustomCallback(Callback):