Repository: pytorch/vision

Issue:
`F.pad` is broken for PIL images and `fill: str`
On the surface `F.pad` for PIL images seems to support str arguments for `fill`:

https://github.com/pytorch/vision/blob/053e7ebd1b50b7a506cfd6e2389043ec05414b1c/torchvision/transforms/functional_pil.py#L157-L158

Internally though, it calls `_parse_fill`

https://github.com/pytorch/vision/blob/053e7ebd1b50b7a506cfd6e2389043ec05414b1c/torchvision/transforms/functional_pil.py#L304-L308

which in most cases relies on fill being numeric. Thus, actually passing a string doesn't work although PIL supports it.

```py
>>> image = PIL.Image.fromarray(torch.randint(0, 256, (256, 256, 3), dtype=torch.uint8).numpy())
>>> F.pad(image, 128, fill="black")
ValueError: invalid literal for int() with base 10: 'black'
>>> PIL.ImageOps.expand(image, border=128, fill="black")
<PIL.Image.Image image mode=RGB size=512x512 at 0x7FA4DD37FBD0>
```

This traces back to #2284 and specifically to

https://github.com/pytorch/vision/blob/37a0d8d68ab2c537d2cda1136d0366874d2fad5f/torchvision/transforms/functional.py#L334-L337

which will makes no sense for `fill: str`. 

Given that #2284 was included from `0.7.0` upwards and we haven't gotten any report yet, my guess is that the functionality is not really used. Since we want to deprecate `fill: str` anyway (https://github.com/pytorch/vision/pull/5621#discussion_r827271131), I would opt to simply remove all mentions of it. Thoughts?



cc @vfdev-5 @datumbox


Discussion:
That's indeed a very convenient bug. We should remove it rather than fix it, I agree. Nevertheless we should consider removing/deprecating string colours from the transforms not only for pad but also for all other transforms as these won't be compatible with the new API for non-PIL images.
In total are two mentions of `fill: str` in `transforms`

https://github.com/pytorch/vision/blob/053e7ebd1b50b7a506cfd6e2389043ec05414b1c/torchvision/transforms/transforms.py#L403

https://github.com/pytorch/vision/blob/053e7ebd1b50b7a506cfd6e2389043ec05414b1c/torchvision/transforms/transforms.py#L598

https://github.com/pytorch/vision/blob/9edd22c8a07a8c99324a73ec4dd6c553e2c8d410/torchvision/transforms/functional.py#L451

https://github.com/pytorch/vision/blob/9edd22c8a07a8c99324a73ec4dd6c553e2c8d410/torchvision/transforms/functional_pil.py#L157-L158

The only "non-pad" reference is on `RandomCrop`, which internally also use `F.pad` and thus is also broken.
How about random erasing's value?
https://github.com/pytorch/vision/blob/9edd22c8a07a8c99324a73ec4dd6c553e2c8d410/torchvision/transforms/transforms.py#L1653-L1654

I suspect that string is not really supported there so worth checking.

There might be more like this. Worth having a deep check on the code to config these are the only references. 
Good catch. I only looked for `fill`. I'll have another look.
Looking again for all `str` mentions, the only additional mention of a `str` representing a numeric value is `RandomErasing` as you found. Still, `value: str` is actually not correct here. It should be `value: Literal["random"]`:

https://github.com/pytorch/vision/blob/9edd22c8a07a8c99324a73ec4dd6c553e2c8d410/torchvision/transforms/transforms.py#L1653-L1656

Resolving patch:
diff --git a/torchvision/transforms/functional.py b/torchvision/transforms/functional.py
--- a/torchvision/transforms/functional.py
+++ b/torchvision/transforms/functional.py
@@ -448,11 +448,11 @@ def pad(img: Tensor, padding: List[int], fill: int = 0, padding_mode: str = "con
             .. note::
                 In torchscript mode padding as single int is not supported, use a sequence of
                 length 1: ``[padding, ]``.
-        fill (number or str or tuple): Pixel fill value for constant fill. Default is 0.
+        fill (number or tuple): Pixel fill value for constant fill. Default is 0.
             If a tuple of length 3, it is used to fill R, G, B channels respectively.
             This value is only used when the padding_mode is constant.
             Only number is supported for torch Tensor.
-            Only int or str or tuple value is supported for PIL Image.
+            Only int or tuple value is supported for PIL Image.
         padding_mode (str): Type of padding. Should be: constant, edge, reflect or symmetric.
             Default is constant.
 
diff --git a/torchvision/transforms/functional_pil.py b/torchvision/transforms/functional_pil.py
--- a/torchvision/transforms/functional_pil.py
+++ b/torchvision/transforms/functional_pil.py
@@ -154,7 +154,7 @@ def pad(
 
     if not isinstance(padding, (numbers.Number, tuple, list)):
         raise TypeError("Got inappropriate padding arg")
-    if not isinstance(fill, (numbers.Number, str, tuple, list)):
+    if not isinstance(fill, (numbers.Number, tuple, list)):
         raise TypeError("Got inappropriate fill arg")
     if not isinstance(padding_mode, str):
         raise TypeError("Got inappropriate padding_mode arg")
diff --git a/torchvision/transforms/transforms.py b/torchvision/transforms/transforms.py
--- a/torchvision/transforms/transforms.py
+++ b/torchvision/transforms/transforms.py
@@ -400,11 +400,11 @@ class Pad(torch.nn.Module):
             .. note::
                 In torchscript mode padding as single int is not supported, use a sequence of
                 length 1: ``[padding, ]``.
-        fill (number or str or tuple): Pixel fill value for constant fill. Default is 0. If a tuple of
+        fill (number or tuple): Pixel fill value for constant fill. Default is 0. If a tuple of
             length 3, it is used to fill R, G, B channels respectively.
             This value is only used when the padding_mode is constant.
             Only number is supported for torch Tensor.
-            Only int or str or tuple value is supported for PIL Image.
+            Only int or tuple value is supported for PIL Image.
         padding_mode (str): Type of padding. Should be: constant, edge, reflect or symmetric.
             Default is constant.
 
@@ -428,7 +428,7 @@ def __init__(self, padding, fill=0, padding_mode="constant"):
         if not isinstance(padding, (numbers.Number, tuple, list)):
             raise TypeError("Got inappropriate padding arg")
 
-        if not isinstance(fill, (numbers.Number, str, tuple, list)):
+        if not isinstance(fill, (numbers.Number, tuple, list)):
             raise TypeError("Got inappropriate fill arg")
 
         if padding_mode not in ["constant", "edge", "reflect", "symmetric"]:
@@ -595,11 +595,11 @@ class RandomCrop(torch.nn.Module):
         pad_if_needed (boolean): It will pad the image if smaller than the
             desired size to avoid raising an exception. Since cropping is done
             after padding, the padding seems to be done at a random offset.
-        fill (number or str or tuple): Pixel fill value for constant fill. Default is 0. If a tuple of
+        fill (number or tuple): Pixel fill value for constant fill. Default is 0. If a tuple of
             length 3, it is used to fill R, G, B channels respectively.
             This value is only used when the padding_mode is constant.
             Only number is supported for torch Tensor.
-            Only int or str or tuple value is supported for PIL Image.
+            Only int or tuple value is supported for PIL Image.
         padding_mode (str): Type of padding. Should be: constant, edge, reflect or symmetric.
             Default is constant.
 


Test patch: