SlowGuess/NeurIPS_2023_OCR huggingface dataset:
CNN-based image classification For image classification using pretrained CNNs such as Resnet and Alexnet, we note that finding a good loss function for equizero is non-trivial. As such, we consider two loss functions 1) negative of the maximum probability as it worked well with the SCAN
(b)
Figure 2: Comparison of zero-shot performance of equizero to equituning and a non-equivariant pretrained model are shown in (a), (b), and (c) for Gridworld, Cartpole, and Acrobot, respectively. Equizero outperforms both equituning and non-equivariant pretrained model. Results over five seeds.
task in §4.1 and 2) the entropy of the output distribution since it correlates with the confidence of the model (Wang et al., 2018). But, equizero with these loss functions performs even worse than equitune. Further, we find that  $\lambda$ -equitune easily outperforms both equitune and equizero.
Here, we provide experimental results for equizero and  $\lambda$ -equitune in §5.1 and §5.2, respectively, for all the applications described in §4. Additional experiments for canonical- $\lambda$ -equitune are provided in §. D.5. The code for this paper is available at https://github.com/basusourya/lambda_equitune.
5.1 Zero-Shot Performance using Equizero
5.1.1 Equizero Reinforcement Learning
Experimental Setting: We first pretrain Deep Q-learning nets (DQNs) for each of the Gridworld, Cartpole, and Acrobot environments using the default architecture from Raffin et al. (2021) with 103k parameters. We pretrained all the models using a learning rate  $10^{-4}$ . We used training time steps as 100k, 100k, and 70k for Gridworld, Cartpole, and Acrobot, respectively. These number of steps were chosen to obtain the best models by varying the time steps from 50k to 100k in multiple of 10k for a fixed seed.
Results and Observations: Fig. 2 show the evaluation performance of equizero and compare it with equituning and non-equivariant pretrained models. We find that equituning performs better than non-equivariant models and equizero outperform both of them. The results are over five seeds.
5.1.2 Fairness in Natural Language Generation
(a)
Figure 3: Plots (a), (b), and (c) show the regard scores for GPT2, EquiGPT2, R-EquiGPT2, EquizeroGPT2, and R-EquizeroGPT2. In equitune, if negativity is present in some demographics, it gets redistributed in the other demographics, which is undesirable. Equitune is only able to debias, whereas, equizero models not only debiases the texts but also makes the regard scores more positive.
Table 2: Zero-shot performance of non-equivariant models, equituned, and equizeroed models for LSTM on SCAN. LSTMs were trained for 200K iterations. We find that equizero outperforms other methods using non-equivariant pretrained models. Results are over three random seeds.
Model	Add Jump	Around Right
Group	Val. Acc.	Test Acc.	Group	Val. Acc.	Test Acc.
LSTM	-	99.1 (0.3)	0.0 (0.0)	-	98.9 (0.7)	0.4 (0.7)
EquiLSTM	Verb	62.5 (1.5)	7.1 (1.3)	Direction	80.5 (4.5)	28.2 (12.9)
EquizeroLSTM	Verb	98.4 (0.9)	75.2 (1.5)	Direction	98.7 (5.8)	81.7 (2.4)
Experimental Setting: We use GPT2 (Radford et al., 2019), with 117M parameters as our pretrained model. We consider the lists of demographic groups [‘man’, ‘woman’], [‘white’, ‘black’], and [‘straight’, ‘gay’]. We compare our method against EquiGPT2 and R-EquiGPT2 (Basu et al., 2023). For the equizero models, we use beam length  $m$  as 5, as described in §C.2. We limit the sentence lengths to be 15 for all models. We generated 500 sentences for each of the models and demographic groups by varying the seeds for both respect and occupation context.
Results and Observations: Fig. 3 and 9 compare the regard scores for all the considered models. We observe that equizero models are not only able to debias among various demographics like 'man' and 'woman', but it also reduces the toxicity/ negativity of the scores. Debiasing is seen from the equality of the scores amongst the demographics considered for equivariance. And reduction in toxicity is observed by noticing that the regard scores are more positive. Like equituning (Basu et al., 2023), equizero models show high quality of generated texts. Sample generations from the demographic groups ['straight', 'gay'] are shown in Tab. 3, 6, 7, and 5 for all the models. Note in Tab. 3 and 6, that even with perfect equivariance, where the word 'straight' simply gets replaced by 'gay', the regard scores are very different. This shows the presence of bias in the regard classifier itself as was also observed by Basu et al. (2023).
5.1.3 Compositional Generalization using Equizero
Experimental Setting: We evaluate the performance of our algorithm on the SCAN Dataset (Gordon et al., 2019) on the Add Jump and Around Right tasks. All the recurrent models (RNN, LSTM, GRU) and their equivariant counterparts contain a single hidden layer of 64 units. For all training processes, we use the Adam optimizer (Kingma & Ba, 2015) and teacher-forcing ratio 0.5 (Williams & Zipser, 1989). For pretraining and finetuning, we use 200k and 10k iterations respectively. For pretraining, we use a learning rate of  $10^{-4}$ , whereas for finetuning, we used learning rates  $10^{-5}$  and  $2 \times 10^{-5}$  for Add Jump and Around Right, respectively. Results are over three seeds.
Observation and Results: Tab. 2 shows that equizero outperforms equituned and non-equivariant models on zero-shot performance with LSTMs. In §D, we observe similar results for RNNs and GRUs in Tab. 9 and 8, respectively. We use gradient estimators discussed in §3.3 for performing few-shot learning using equizero. For few-shot learning, we find in Fig. 10 in §D that equizero is competitive with equitune for small iterations, but as the number of steps increase, equitune is the better algorithm. This is expected since the gradients are computable for equitune, but only approximated in equizero. Tab. 10, 11, and 12 in §D provide the results for finetuning using equitune and equizero for 10k iterations. We find the equitune is slightly better when finetuning for 10k iterations. This shows equizero is better for zero-shot and few-shot learning, but for large iterations, equitune is preferable.
5.2 Zero-Shot and Finetuning Performance using  $\lambda$ -Equitune
5.2.1 Equi/Invariant Image Classification using CLIP
Experimental Setting: We first perform zero-shot image classification on Imagenet-V2 and CIFAR100 using CLIP. We use two transforms, random  $90^{\circ}$  rotations and flips, for testing their robustness. We encode class labels using the 80 text templates provided in Radford et al. (2021) $^{2}$ .
2Obtained from https://github.com/openai/CLIP
(b)
Figure 4: In (a) note that zero-shot performance of CLIP drops significantly when we add random rotations or flips to the input. This trend is seen across all image encoders, i.e. ResNet (RN50 and RN101) and ViT (ViT-B/32 and ViT-B/16). In (b) and (c) we show classification results on Imagenet-V2 with random  $90^{\circ}$  rotations and flips, respectively. We observe equizero outperform equitune and original CLIP for all image encoders.
We then evaluate finetuning capabilities of equitune, equizero, and  $\lambda$ -equitune on CIFAR100 with random  $90^{\circ}$  rotations. Here we choose  $\lambda$  to be a two-layered feed-forward network, which is described in detail along with learning rates used, in §D.4. The input to this  $\lambda$ -network are the features from a frozen CLIP encoder. First, the  $\lambda$ -network is trained with the model kept frozen. Then, while finetuning the model, the  $\lambda$  network is frozen. We show results for both Resnet and ViT backbone trained for 1000, 2000, 3000, 4000 finetuning steps.
Results and Observations: Fig. 4 shows test accuracies for Imagenet-V2 with random  $90^{\circ}$  rotations and flips. We observe that the pretrained model's performance reduces drastically when transformations are applied to the dataset. Whereas both equitune and equizero are relatively robust to such transformations. Moreover, equizero outperforms both equitune and the pretrained model. Similar observations are made for CIFAR100 in Fig. 8 in  $\S D$ .
In Fig. 5a and 11a we plot the test accuracies of  $\lambda$ -equitune on CIFAR100 for both variants of Resnet and ViT backbones. We observe that  $\lambda$ -equitune performs better than both equitune and equizero (with finetuning) on Resnets. On ViT-B/16, we observe that  $\lambda$ -equitune easily outperforms both equitune and equizero (with finetuning). On ViT-B/32, we find that  $\lambda$ -equitune outperforms equitune but equizero outperforms both equitune and  $\lambda$ -equitune. Thus,  $\lambda$ -equitune performs competitively with equizero, even in applications where good loss functions are known.
5.2.2 Equi/Invariant Image Classification using Pretrained CNNs
Experimental Setting: We now evaluate the performance of  $\lambda$ -equitune on rot90-CIFAR10 dataset. Here, each image is rotated randomly by a multiple of  $90^{\circ}$ . We use pretrained Alexnet and Resnet, trained extensively over CIFAR100. For the  $\lambda$ -equituning, we choose  $\lambda$  as a two layered feed-forward network with a hidden layer of dimension 100 with input as features extracted by this pretrained Alexnet or Resnet. Along with  $\lambda$ , we also perform linear probing wherein last two layers for the classification problem is being learned using a learning rate of  $10^{-3}$ .
Observation and Results: In Fig. 5b and 11b we see that  $\lambda$ -equitune outperforms equitune and equizero. Moreover, equizero performs even worse than equitune. This trend is consistent across both Alexnet and Resnet pretrained modules.
6 Limitations, Societal Impact, and Conclusion
Limitations and Societal Impact: Our results focus on finite groups; extension to continuous groups requires further work in parameterization of continuous groups (cf. Benton et al. (2020)) or group decomposition (cf. Basu et al. (2021); Maile et al. (2023)). Our work on fairness in NLG aims to debias foundation models and thereby lead to positive societal impact. But we use equality and neutral sets obtained from previous human-made works. We believe there is scope for optimizing the design of such sets using neural networks for more general demographic groups.
Conclusion: We present  $\lambda$ -equitune and its special case equizero that outperform equitune on equivariant zero-shot and finetuning tasks. We show that both methods are equivariant, universal,
(b)
Figure 5: In (a) and (b), we plot accuracies of pretrained CLIP and Resnet, respectively, for equitune, equizero, and  $\lambda$ -equitune. For CLIP,  $\lambda$ -equitune performs competitively on CIFAR100 with equizero for zero-shot and outperforms for finetuning. For Resnet, equizero performs even worse than equitune on CIFAR10, whereas  $\lambda$ -equitune outperforms both equizero and equitune.
are computationally efficient, and are easy to implement. Equizero performs well when good proxy loss functions are available for downstream tasks, which we show is easy to find across several diverse tasks.  $\lambda$ -equitune outperforms equitune and is competitive with equizero without the need for any proxy loss. We consider diverse tasks: i) deep Q-learning, ii) fairness in NLG, iii) compositional generalization in languages, iv) CLIP-based classification, and v) CNN-based classification. Experimental results validate the superiority of  $\lambda$ -equitune and equizero over equitune on zero-shot and finetuning tasks.
Discussion with Moulik Choraria on the  $\lambda$ -equitune finetuning for CLIP is appreciated. A portion of the work was supported by the Department of Energy (DOE) award (DE-SC0012704). This work was also supported in part by ZJU-UIUC Joint Research Center Project No. DREMES 202003, funded by Zhejiang University.
Atzmon, M., Nagano, K., Fidler, S., Khamis, S., and Lipman, Y. Frame averaging for equivariant shape space learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 631-641, 2022.
Basu, S., Magesh, A., Yadav, H., and Varshney, L. R. Autoequivariant network search via group decomposition. arXiv:2104.04848, 2021.
Basu, S., Gallego-Posada, J., Viganò, F., Rowbottom, J., and Cohen, T. Equivariant mesh attention networks. Transactions on Machine Learning Research, 2022.
Basu, S., Sattigeri, P., Ramamurthy, K. N., Chenthamarakshan, V., Varshney, K. R., Varshney, L. R., and Das, P. Equi-tuning: Group equivariant fine-tuning of pretrained models. In AAAI Conference on Artificial Intelligence, pp. 6788-6796, 2023.
Bengio, Y., Léonard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv:1308.3432, 2013.
Benton, G., Finzi, M., Izmailov, P., and Wilson, A. G. Learning invariances in neural networks from training data. Advances in Neural Information Processing Systems, 33:17605-17616, 2020.
Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. OpenAI Gym, 2016.