Repository: pytorch/vision

Issue:
The labels in the QMNIST dataset are all equal to 0.
### 🐛 Describe the bug

The labels in the QMNIST dataset are all equal to 0.
```python
im_transform = T.Compose([
    T.ToTensor()
])

qmnist_train = torchvision.datasets.QMNIST('data/', train=True, download=True, transform=im_transform)

# the first image is a '5'
plt.imshow(qmnist_train[0][0][0])
# prints '0'
print(qmnist_train[0][1])

# the following never outputs anything
[print(qmnist_train[i][1]) for i in range(len(qmnist_train)) if qmnist_train[i][1] != 0]
```

### Versions

```
Collecting environment information...
PyTorch version: 1.13.1
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.10.8 (main, Nov 24 2022, 14:13:03) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-57-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.6.124
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 980 Ti
Nvidia driver version: 525.60.11
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] geotorch==0.3.0
[pip3] numpy==1.23.5
[pip3] torch==1.13.1
[pip3] torchaudio==0.13.1
[pip3] torchvision==0.14.1
[conda] blas                      1.0                         mkl  
[conda] ffmpeg                    4.3                  hf484d3e_0    pytorch
[conda] geotorch                  0.3.0                    pypi_0    pypi
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0           py310h7f8727e_0  
[conda] mkl_fft                   1.3.1           py310hd6ae3a3_0  
[conda] mkl_random                1.2.2           py310h00e6091_0  
[conda] numpy                     1.23.5          py310hd5efca6_0  
[conda] numpy-base                1.23.5          py310h8e6c178_0  
[conda] pytorch                   1.13.1          py3.10_cuda11.6_cudnn8.3.2_0    pytorch
[conda] pytorch-cuda              11.6                 h867d48c_1    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.13.1              py310_cu116    pytorch
[conda] torchvision               0.14.1              py310_cu116    pytorch
```

cc @pmeier


Discussion:
Can reproduce.
This is caused by a faulty byte flip in

https://github.com/pytorch/vision/blob/7dc5e5bd60b55eb4e6ea5c1265d6dc7b17d2e917/torchvision/datasets/mnist.py#L525-L528

which was introduced in #4598. Due to the extra `num_bytes_per_value > 1` this can only trigger for QMNIST labels, which is why we didn't detect it before.
Given the amount of discussion in #4882 and the final conclusion 

https://github.com/pytorch/vision/blob/7dc5e5bd60b55eb4e6ea5c1265d6dc7b17d2e917/torchvision/prototype/utils/_internal.py#L94-L97

I don't think byte flipping is possible in PyTorch. `Tensor.flip` just flips the elements along a dimension, but leaves the values as is:

```pycon
>>> t = torch.arange(4)
>>> t
tensor([0, 1, 2, 3])
>>> t.flip(0)
tensor([3, 2, 1, 0])
```

I think we should revert #4598 unless something happened in the past year that allows us to do byte flipping from PyTorch easily. cc @NicolasHug 
Ok, byte flipping, i.e. changing the endianness, is possible with PyTorch, by viewing the data through the `torch.uint8` dtype:

```py
if needs_byte_reversal:
    parsed = (
        parsed.contiguous()
        .view(torch.uint8)
        .view(parsed.numel(), num_bytes_per_value)
        .flip(1)
        .flatten()
        .view(torch_type)
    )
```

Thanks for the tip @lezcano!
Thanks a lot for investigating @pmeier, and @Foobin for the report.

Perf isn't critical at this stage of the dataset loading, so we can go with either solution (revert to `np`, or use the native pytorch code above).

Any pref Philip?
Minimally leaning towards the PyTorch only solution, since we tried to phase out `numpy` as much as possible for our prototype datsets (see #4882). But I'm fine with either.
I know you guys mentioned that perf is not an issue here, but the PyTorch version should be much faster on CPU due to @vfdev-5 's recent vectorisation of `flip`. 
Good to know! If we can confirm that with some quick benchmarks, we could also potentially simplify the code @pmeier mentioned above https://github.com/pytorch/vision/issues/7079#issuecomment-1379976150 ?

Resolving patch:
diff --git a/torchvision/datasets/mnist.py b/torchvision/datasets/mnist.py
--- a/torchvision/datasets/mnist.py
+++ b/torchvision/datasets/mnist.py
@@ -12,7 +12,7 @@
 import torch
 from PIL import Image
 
-from .utils import check_integrity, download_and_extract_archive, extract_archive, verify_str_arg
+from .utils import _flip_byte_order, check_integrity, download_and_extract_archive, extract_archive, verify_str_arg
 from .vision import VisionDataset
 
 
@@ -519,13 +519,12 @@ def read_sn3_pascalvincent_tensor(path: str, strict: bool = True) -> torch.Tenso
     torch_type = SN3_PASCALVINCENT_TYPEMAP[ty]
     s = [get_int(data[4 * (i + 1) : 4 * (i + 2)]) for i in range(nd)]
 
-    num_bytes_per_value = torch.iinfo(torch_type).bits // 8
-    # The MNIST format uses the big endian byte order. If the system uses little endian byte order by default,
-    # we need to reverse the bytes before we can read them with torch.frombuffer().
-    needs_byte_reversal = sys.byteorder == "little" and num_bytes_per_value > 1
     parsed = torch.frombuffer(bytearray(data), dtype=torch_type, offset=(4 * (nd + 1)))
-    if needs_byte_reversal:
-        parsed = parsed.flip(0)
+
+    # The MNIST format uses the big endian byte order, while `torch.frombuffer` uses whatever the system uses. In case
+    # that is little endian and the dtype has more than one byte, we need to flip them.
+    if sys.byteorder == "little" and parsed.element_size() > 1:
+        parsed = _flip_byte_order(parsed)
 
     assert parsed.shape[0] == np.prod(s) or not strict
     return parsed.view(*s)
diff --git a/torchvision/datasets/utils.py b/torchvision/datasets/utils.py
--- a/torchvision/datasets/utils.py
+++ b/torchvision/datasets/utils.py
@@ -520,3 +520,9 @@ def _read_pfm(file_name: str, slice_channels: int = 2) -> np.ndarray:
     data = np.flip(data, axis=1)  # flip on h dimension
     data = data[:slice_channels, :, :]
     return data.astype(np.float32)
+
+
+def _flip_byte_order(t: torch.Tensor) -> torch.Tensor:
+    return (
+        t.contiguous().view(torch.uint8).view(*t.shape, t.element_size()).flip(-1).view(*t.shape[:-1], -1).view(t.dtype)
+    )


Test patch:
diff --git a/test/test_datasets_utils.py b/test/test_datasets_utils.py
--- a/test/test_datasets_utils.py
+++ b/test/test_datasets_utils.py
@@ -7,7 +7,9 @@
 import zipfile
 
 import pytest
+import torch
 import torchvision.datasets.utils as utils
+from common_utils import assert_equal
 from torch._utils_internal import get_file_path_2
 from torchvision.datasets.folder import make_dataset
 from torchvision.datasets.utils import _COMPRESSED_FILE_OPENERS
@@ -215,6 +217,24 @@ def test_verify_str_arg(self):
         pytest.raises(ValueError, utils.verify_str_arg, 0, ("a",), "arg")
         pytest.raises(ValueError, utils.verify_str_arg, "b", ("a",), "arg")
 
+    @pytest.mark.parametrize(
+        ("dtype", "actual_hex", "expected_hex"),
+        [
+            (torch.uint8, "01 23 45 67 89 AB CD EF", "01 23 45 67 89 AB CD EF"),
+            (torch.float16, "01 23 45 67 89 AB CD EF", "23 01 67 45 AB 89 EF CD"),
+            (torch.int32, "01 23 45 67 89 AB CD EF", "67 45 23 01 EF CD AB 89"),
+            (torch.float64, "01 23 45 67 89 AB CD EF", "EF CD AB 89 67 45 23 01"),
+        ],
+    )
+    def test_flip_byte_order(self, dtype, actual_hex, expected_hex):
+        def to_tensor(hex):
+            return torch.frombuffer(bytes.fromhex(hex), dtype=dtype)
+
+        assert_equal(
+            utils._flip_byte_order(to_tensor(actual_hex)),
+            to_tensor(expected_hex),
+        )
+
 
 @pytest.mark.parametrize(
     ("kwargs", "expected_error_msg"),