Human-in-the-loop mixup
Katherine M. Collins*, Umang Bhatt, Weiyang Liu, Vihari Piratla, Bradley Love, Adrian Weller*kmc61@cam.ac.uk
HSE 1: What do humans believe
matches a given ?

Motivation● Synthetic data is increasingly powering advances inmachine learning (ML)1,2,3● Not always clear whether human perceptual
judgments of synthetically-generated exampes matchthe generative process used to create them
● First step: consider
synthetic examples
used in mixup4

Elicitation (N = 70)● Elicit perceived 50/50 point over 249
pairs of mixed CIFAR-1011 examples● Employ different elicitation interfaces○ Construct: press arrow keys to selectmixed image○ Select-Shuffled: choose from ashuffled set of mixed images■ Controls for order effects

Why care about humanpercepts of synthetic data
generation?
● Further ensure model reliability
and trustworthiness● Realigning may improvedownstream model performance● Help guard against gamificationand manipulation● Can we align the labeling of mixed images to human perception tolearn better category boundaries?

Set-upFig 3: Genericelicitation paradigm

Findings● In general, humans recover 50/50 mix● But nuanced picture at individual-level
suggests misalignment● Decent agreement across interfaces

Fig 1: Example mixed image

Learning with Human Relabelings● Augment training set with mixed
images and constructed labels● Explore levering averaged
human relabelngs without
confidence● And leveraging elicited humanconfidence (⍵) to smooth
between a uniform distribution
and the averaged humanrelabelingFig 4: Individual
endorsements of theperceived 50/50 point

Why mixup?
● Simple data generation: access to “ground truth” parameterization● Powerful technique and popular baseline5,6,7● Cognitive neuroscience suggests likely misalignment8,9,10

Learning with Automatic Label Policy
Grounded in Human Inferences

Fig 5: Example consensus misalignment of 50/50

HSE 2: Conditioned on , what do
humans perceive to be a good as ?

Problem Setting● mixup4 is an effective regularizer, which trains on linear
combinations of examples● Examples constructed via data and label mixing policies

Eq 1: Data mixing policy

Elicitation (N = 81)● 2070 mixed images● Tell people the underlying labels● Ask to infer the mixing coefficient, andprovide their confidence in estimateFindings● Discrepancies elucidated between humans’internal models of synthetically-generated
data vs. label mixing policy used in mixup● In aggregate + individual-level misalignment

Eq 2: Label mixing policy

Fig 6: Genericelicitation
paradigm

Table 1: Evaluating on
CIFAR-10H11,12 holdout, withand without human feedback● How can we go beyond the constraint of finite human labelings for an
infinite set of possible synthetic examples?
● Category boundaries have diverse structures, many non-linear — canwe leverage capture this structure in mixup label policy?

Set-upFig 9: Example inferred mixing coefficient for category pairs● Fit logistic function to each category pair
● Compare learning with transformed
mixing coefficient against
classicial, full mixup4
Table 2: Comparing automated full relabelingschemes

Takeaways

Fig 2: Synthetic data generative process used in mixup● Our Approach: Study human perception of thegenerative process through two human subject
experiment (HSE) paradigms

References
1.
2.
3.
4.
5.
6.

Celso M. de Melo, Antonio Torralba, Leonidas Guibas, James DiCarlo, Rama Chellappa, and Jessica Hodgins. 2022. Next-generation deep learning based on simulators and synthetic data. Trends in Cognitive Sciences 26, 2 (2022),
174–187. https://doi.org/10.1016/j.tics.2021.11.008
James Jordon, Lukasz Szpruch, Florimond Houssiau, Mirko Bottarelli, Giovanni Cherubin, Carsten Maple, Samuel N Cohen, and Adrian Weller. 2022. Synthetic Data–what, why and how? arXiv preprint arXiv:2205.03257 (2022).
Shorten, Connor, and Taghi M. Khoshgoftaar. "A survey on image data augmentation for deep learning." Journal of big data 6.1 (2019): 1-48.
Zhang, H.; Ciss é , M.; Dauphin, Y. N.; and Lopez-Paz, D. 2017. mixup: Beyond Empirical Risk Minimization. CoRR, abs/1710.09412.
Thulasidasan, S.; Chennupati, G.; Bilmes, J.; Bhattacharya, T.; and Michalak, S. 2019. On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks.
Linjun Zhang, Zhun Deng, Kenji Kawaguchi, and James Zou. 2022. When and How Mixup Improves Calibration. In International Conference on Machine Learning (ICML).

Fig 7: Aggregate human-inferred mixing coeff(blue) vs. mixup (red)

7.
8.
9.
10.
11.
12.
13.

Fig 8: Example synthetic image-level
relabelings● Human percepts not consistently aligned with data generation used in mixup○ When considering both data and label mixing policy● Relabeling with human percepts, particularly when leveraging humanconfidence, has potential to improve downstream model performance

Dan Hendrycks, Andy Zou, Mantas Mazeika, Leonard Tang, Bo Li, Dawn Song, and Jacob Steinhardt. 2022. Pixmix: Dreamlike pictures comprehensively improve safety measures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16783–16792.
Jonathan R Folstein, Thomas J Palmeri, and Isabel Gauthier. 2013. Category learning increases discriminability of relevant object dimensions in visual cortex. Cerebral Cortex 23, 4 (2013), 814–823
Robert L Goldstone and Andrew T Hendrickson. 2010. Categorical perception. Wiley Interdisciplinary Reviews: Cognitive Science 1, 1 (2010), 69–78.
Stevan Harnad. 2003. Categorical perception. (2003).
Alex Krizhevsky. 2009. Learning multiple layers of features from tiny images. Technical Report.
Peterson, J. C.; Battleday, R. M.; Griffiths, T. L.; and Russakovsky, O. 2019. Human uncertainty makes classification more robust. CoRR, abs/1908.07086
Battleday, R. M.; Peterson, J. C.; and Griffiths, T. L. 2019. Capturing human categorization of natural images at scale by combining deep networks and cognitive models. CoRR, abs/1904.12690

Presented at The AAAI 2023 Workshop on Representation Learning for
Responsible Human-Centric AI