SlowGuess/NeurIPS_2023_OCR huggingface dataset:
DAVIS To train, test and compare these models, we use the DAVIS dataset [16], which was originally designed as a benchmark for video object segmentation. Image sequences in this dataset contain diverse motion of scenes and objects (eg., with fixed or moving camera, and objects moving at different speeds and directions), which make next frame prediction challenging. Each clip is sampled at 25 frames per second, and is approximately 3 seconds long. The set is subdivided into 60 training videos (4741 frames) and 30 test videos (2591 frames). We pre-processed the data, converting all frames to monochrome luminance values, and scaling their range to the interval  $[-1, 1]$ . Frames are cropped to a  $256 \times 256$  central region, where most of the motion tends to occur, and then spatially down-sampled to  $128 \times 128$  pixels.
VanHateren We also consider a smaller video dataset consisting in footage of animals in the wild [48] which contains a variety of motions (animals in the scene, camera motion, etc.) and occlusions. The missing band at the top the frame is cropped, reducing the image size from  $128 \times 128$  pixels to  $112 \times 128$  pixels. The dataset is standardized to zero mean and unit variance, it is split into snippets of 11 frames, 292 snippets are used for training and 33 for testing. There is no spatial downsampling or whitening.
Boundary handling The computation of this prediction error is restricted to the center of the image because moving content that enters from outside the video frame is inherently unpredictable. Specifically, we trim a 17-pixel strip from each side, yielding frames of size  $94 \times 94$  pixels for the DAVIS dataset and  $78 \times 94$  for the VanHateren dataset. Convolutions are computed with zero padding so that the outputs have the same size as inputs (the only exception is for the plain Polar Predictor, shown in Figure 6, where valid convolutions were performed).
Training procedure We assume the temporal evolution of natural signals to be sufficiently and appropriately diverse for training, and do not apply any additional data augmentation procedures. We train on brief temporal segments containing 11 frames (which allows for prediction of 9 frames), and process these in batches of size 4. We train each model for 200 epochs on DAVIS using the Adam optimizer [49] with default parameters and a learning rate of  $3 \cdot 10^{-4}$ . The learning rate is automatically halved when the test loss plateaus. In the CNN, we use batch normalization before every half-wave rectification, rescaling by the standard deviation of channel coefficients (but with no additive bias terms).
Additional dataset Note that the PSNR is the logarithm of the MSE in units of the signal:  $\mathrm{PSNR} = 10\log_{10}(\mathrm{MAX}^2/\mathrm{MSE})$ , where MAX is the maximum possible pixel value of the image. A PSNR of 0dB means that the squared peak signal and the mean squared error are equal; a PSNR of 10dB (resp. 20dB) means that the squared peak signal is 10 times (resp. 100 times) bigger than the MSE.
Table 2: Performance comparison. Prediction error computed on the VanHateren dataset. Values indicate mean (standard deviation) of Peak Signal to Noise Ratio (PSNR in dB) computed over 10 random seeds.
Method
split	Copy	cMC	SPyr	mPP	mQP	CNN	Unet
train	27.10	28.56	29.50	30.16 (0.02)	30.28 (0.04)	31.16 (0.10)	31.52 (0.42)
test	26.41	27.93	28.83	29.11 (0.01)	29.07 (0.04)	29.09 (0.05)	29.26 (0.06)
Computational costs The polar predictor described in this paper is lightweight: it runs rapidly and contains few parameters (two orders of magnitude less than CNN and Unet). The polar predictor is designed as an online method that could be applied to streaming data. Parameter counts, as well as training and inference time are reported in Table 3.
Notice that the Unet is also very efficient: it runs fast (most of the computation is applied to spatially downsampled coefficients). Note that the Quadratic Predictor (mQP) is slower. This model was
Table 3: Number of trainable parameters and run times. Mean training time (standard deviation) for 200 epochs on the VanHateren dataset (mean in min:sec, std in sec). Mean inference time (standard deviation) for a single video snippet containing 11 frames of size  $112 \times 128$  pixels (mean in ms, std in  $\mu s$ ). Training and inference time are computed on a NVIDIA A100 GPU.
dataset	Method
mPP	mQP	CNN	Unet
# parameters	7,744	15,776	666,496	591,041
training time	10:24 (20)	46:14 (29)	32:17 (50)	11:14 (17)
inference time	9.71 (11)	19.8 (8)	9.57 (8.4)	2.7 (8.2)
developed to suggest a connection with physiology (each element recapitulates know functional building blocks of primate visual physiology), not to improve performance or efficiency.
Decoupling analysis and synthesis When decoupling the analysis and synthesis transforms (ie. unting the weights), a Polar Predictor (not multiscale) achieved a similar prediction performance on the VanHateren dataset (tied: train 29.87/ test 28.83; vs. untied: train 29.99 / test 28.87 dB). The projection and basis vectors (ie. the filters in the analysis and synthesis transforms) align as training progresses.