Repository: wandb/wandb

Issue:
fastai callback fails with images in gan example
`wandb --version && python --version && uname`

* Weights and Biases version: 0.8.21
* Python version: 3.7.4
* Operating System: linux

### Description

```
learn_critic.fit_one_cycle(5, 1e-3, callbacks=[WandbCallback(learn_critic,
                                                             log='all', input_type='images', monitor='accuracy_thresh_expand')])
```
causes
```
  File "train.py", line 46, in <module>
    log='all', input_type='images', monitor='accuracy_thresh_expand')])
  File "/home/lukas/.pyenv/versions/3.7.4/lib/python3.7/site-packages/fastai/train.py", line 23, in fit_one_cycle
    learn.fit(cyc_len, max_lr, wd=wd, callbacks=callbacks)
  File "/home/lukas/.pyenv/versions/3.7.4/lib/python3.7/site-packages/fastai/basic_train.py", line 200, in fit
    fit(epochs, self, metrics=self.metrics, callbacks=self.callbacks+callbacks)
  File "/home/lukas/.pyenv/versions/3.7.4/lib/python3.7/site-packages/fastai/basic_train.py", line 108, in fit
    if cb_handler.on_epoch_end(val_loss): break
  File "/home/lukas/.pyenv/versions/3.7.4/lib/python3.7/site-packages/fastai/callback.py", line 317, in on_epoch_end
    self('epoch_end', call_mets = val_loss is not None)
  File "/home/lukas/.pyenv/versions/3.7.4/lib/python3.7/site-packages/fastai/callback.py", line 251, in __call__
    for cb in self.callbacks: self._call_and_update(cb, cb_name, **kwargs)
  File "/home/lukas/.pyenv/versions/3.7.4/lib/python3.7/site-packages/fastai/callback.py", line 241, in _call_and_update
    new = ifnone(getattr(cb, f'on_{cb_name}')(**self.state_dict, **kwargs), dict())
  File "/home/lukas/client/wandb/fastai/__init__.py", line 143, in on_epoch_end
    pred = self.learn.predict(x)
  File "/home/lukas/.pyenv/versions/3.7.4/lib/python3.7/site-packages/fastai/basic_train.py", line 384, in predict
    y = ds.y.reconstruct(pred, x) if has_arg(ds.y.reconstruct, 'x') else ds.y.reconstruct(pred)
  File "/home/lukas/.pyenv/versions/3.7.4/lib/python3.7/site-packages/fastai/data_block.py", line 386, in reconstruct
    return Category(t, self.classes[t])
IndexError: list index out of range

```

[example.zip](https://github.com/wandb/client/files/4087747/example.zip)



Discussion:
The callback depends on learn_critic.predict which does not seem to be supported with this type of data.

```
x, y= learn_critic.data.valid_ds[0]
learn_critic.predict(x)
```

However `pred_batch` is supported

```
x, y= learn_critic.data.valid_ds[0]
learn_critic.pred_batch([x])
```

I'm gonna see if we can use this method instead as a workaround.
This is actually difficult to implement as several methods from fastai are involved in the prediction and it's not fully supported (with `show` functions) for all data types.
Should we just add a "try" block and log a warning when it's not supported?
If we can capture a class of errors smaller than Exception that's ideal

On Tue, Jan 21, 2020 at 4:16 PM Boris Dayma <notifications@github.com>
wrote:

> This is actually difficult to implement as several methods from fastai are
> involved in the prediction and it's not fully supported (with show
> functions) for all data types.
> Should we just add a "try" block and log a warning when it's not supported?
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/wandb/client/issues/783?email_source=notifications&email_token=AAAAAELCC2W3T4INAMV2AJDQ65QZLA5CNFSM4KJJUJJ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJRJFFA#issuecomment-576885396>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAAAAEPKFWXKD2PKR66G4PLQ65QZLANCNFSM4KJJUJJQ>
> .
>


Resolving patch:
diff --git a/wandb/fastai/__init__.py b/wandb/fastai/__init__.py
--- a/wandb/fastai/__init__.py
+++ b/wandb/fastai/__init__.py
@@ -11,7 +11,7 @@
         learn = Learner(data, ..., callback_fns=WandbCallback)
         learn.fit(epochs)
     ```
-    
+
     Custom parameters can be given using functools.partial::
 
     ```
@@ -64,7 +64,7 @@ class WandbCallback(TrackerCallback):
         predictions (int): number of predictions to make if input_type is set and validation_data is None.
         seed (int): initialize random generator for sample predictions if input_type is set and validation_data is None.
     """
-    
+
     # Record if watch has been called previously (even in another instance)
     _watch_called = False
 
@@ -79,7 +79,6 @@ def __init__(self,
                  predictions=36,
                  seed=12345):
 
-
         # Check if wandb.init has been called
         if wandb.run is None:
             raise ValueError(
@@ -136,61 +135,16 @@ def on_epoch_end(self, epoch, smooth_loss, last_metrics, **kwargs):
                 with self.model_path.open('wb') as model_file:
                     self.learn.save(model_file)
 
-        # Log sample predictions
+        # Log sample predictions if learn.predict is available
         if self.validation_data:
-            pred_log = []
-
-            for x, y in self.validation_data:
-                pred = self.learn.predict(x)
-
-                # scalar -> likely to be a category
-                if not pred[1].shape:
-                    pred_log.append(
-                        wandb.Image(
-                            x.data,
-                            caption='Ground Truth: {}\nPrediction: {}'.format(
-                                y, pred[0])))
-
-                # most vision datasets have a "show" function we can use
-                elif hasattr(x, "show"):
-                    # log input data
-                    pred_log.append(
-                        wandb.Image(x.data, caption='Input data', grouping=3))
-
-                    # log label and prediction
-                    for im, capt in ((pred[0], "Prediction"),
-                                     (y, "Ground Truth")):
-                        # Resize plot to image resolution
-                        # from https://stackoverflow.com/a/13714915
-                        my_dpi = 100
-                        fig = plt.figure(frameon=False, dpi=my_dpi)
-                        h, w = x.size
-                        fig.set_size_inches(w / my_dpi, h / my_dpi)
-                        ax = plt.Axes(fig, [0., 0., 1., 1.])
-                        ax.set_axis_off()
-                        fig.add_axes(ax)
-
-                        # Superpose label or prediction to input image
-                        x.show(ax=ax, y=im)
-                        pred_log.append(wandb.Image(fig, caption=capt))
-                        plt.close(fig)
-
-                # likely to be an image
-                elif hasattr(y, "shape") and (
-                    (len(y.shape) == 2) or
-                    (len(y.shape) == 3 and y.shape[0] in [1, 3, 4])):
-
-                    pred_log.extend([
-                        wandb.Image(x.data, caption='Input data', grouping=3),
-                        wandb.Image(pred[0].data, caption='Prediction'),
-                        wandb.Image(y.data, caption='Ground Truth')
-                    ])
-
-                # we just log input data
-                else:
-                    pred_log.append(wandb.Image(x.data, caption='Input data'))
-
-            wandb.log({"Prediction Samples": pred_log}, commit=False)
+            try:
+                self._wandb_log_predictions()
+            except FastaiError as e:
+                wandb.termwarn(e.message)
+                self.validation_data = None  # prevent from trying again on next loop
+            except Exception as e:
+                wandb.termwarn("Unable to log prediction samples.\n{}".format(e))
+                self.validation_data=None  # prevent from trying again on next loop
 
         # Log losses & metrics
         # Adapted from fast.ai "CSVLogger"
@@ -212,3 +166,67 @@ def on_train_end(self, **kwargs):
                     self.learn.load(model_file, purge=False)
                     print('Loaded best saved model from {}'.format(
                         self.model_path))
+
+    def _wandb_log_predictions(self):
+        "Log prediction samples"
+
+        pred_log = []
+
+        for x, y in self.validation_data:
+            try:
+                pred=self.learn.predict(x)
+            except:
+                raise FastaiError('Unable to run "predict" method from Learner to log prediction samples.')
+
+            # scalar -> likely to be a category
+            if not pred[1].shape:
+                pred_log.append(
+                    wandb.Image(
+                        x.data,
+                        caption='Ground Truth: {}\nPrediction: {}'.format(
+                            y, pred[0])))
+
+            # most vision datasets have a "show" function we can use
+            elif hasattr(x, "show"):
+                # log input data
+                pred_log.append(
+                    wandb.Image(x.data, caption='Input data', grouping=3))
+
+                # log label and prediction
+                for im, capt in ((pred[0], "Prediction"),
+                                 (y, "Ground Truth")):
+                    # Resize plot to image resolution
+                    # from https://stackoverflow.com/a/13714915
+                    my_dpi = 100
+                    fig = plt.figure(frameon=False, dpi=my_dpi)
+                    h, w = x.size
+                    fig.set_size_inches(w / my_dpi, h / my_dpi)
+                    ax = plt.Axes(fig, [0., 0., 1., 1.])
+                    ax.set_axis_off()
+                    fig.add_axes(ax)
+
+                    # Superpose label or prediction to input image
+                    x.show(ax=ax, y=im)
+                    pred_log.append(wandb.Image(fig, caption=capt))
+                    plt.close(fig)
+
+            # likely to be an image
+            elif hasattr(y, "shape") and (
+                (len(y.shape) == 2) or
+                    (len(y.shape) == 3 and y.shape[0] in [1, 3, 4])):
+
+                pred_log.extend([
+                    wandb.Image(x.data, caption='Input data', grouping=3),
+                    wandb.Image(pred[0].data, caption='Prediction'),
+                    wandb.Image(y.data, caption='Ground Truth')
+                ])
+
+            # we just log input data
+            else:
+                pred_log.append(wandb.Image(x.data, caption='Input data'))
+
+            wandb.log({"Prediction Samples": pred_log}, commit=False)
+
+
+class FastaiError(wandb.Error):
+    pass


Test patch: