Neural Representations in Multi-Task Learning guided by Task-Dependent Contexts
Authors: Santiago Galella
Salva Ardid
Venue: Submitted to ICLR 2023 (2023)
Keywords: Learning Representations
Neural Geometry
Context-Dependent Decision Making
Attention Mechanisms

Abstract: The ability to switch between tasks effectively in response to external stimuli is a hallmark of cognitive control. Our brain is able to filter and to integrate external information to accomplish goal-directed behavior. Task switching occurs rapidly and efficiently, allowing us to perform multiple tasks with ease. In a similar way, deep learning models can be tailored to exhibit multi-task capabilities and achieve high performance across domains. Still, understanding how neural networks make predictions is crucial in many real-world applications.

In this study, we delve into neural representations learned by multi-tasking architectures. Concretely, we compare individual and parallel networks with task switching networks. Task-switching networks leverage task-dependent contexts to learn disentangled representations without hurting the overall task accuracy. We show that task-switching networks operate in an intermediate regime between individual and parallel. In addition, we show that shared representations are produced by the emergence neurons encoding multiple tasks. Furthermore, we study the role of contexts across network processing and show its role at aligning the task with the relevant features. Finally, we investigate how the magnitude of contexts affects the performance in task-switching networks.

Under review conference paper ICLR 2023
NEURAL REPRESENTATIONS IN MULTI-TASK LEARNING GUIDED BY TASK-DEPENDENT CONTEXTS
Anonymous authors Paper under double-blind review
ABSTRACT
The ability to switch between tasks effectively in response to external stimuli is hallmark of cognitive control. Our brain is able to filter and integrate external in- formation to accomplish goal-directed behavior. Task switching occurs rapidly and efficiently. allowing us to perform multiple tasks with ease. In similar way, artificial neural networks can be tailored to exhibit multi-task task capabilities and achieve high performance across domains.
terms of explainability, un- derstanding how neural networks make predictions is crucial in many real-world applications, for instance, in guiding clinical decisions.
In this study, we delve into neural representations learned by multi-tasking ar- chitectures. Concretely, we compare individual and parallel networks with lask- switching networks. saik-switching networks leverage task tusk-dependent contexts to learn disentangled representations without hurting the overall task accuracy. We show that task-switching networks operate in an intermediate regime between in- dividual and parallel networks. In addition, we show that shared representations are produced by the emergence neurons encoding multiple tasks. Furthermore, we network processing and show its role at aligning
study the role of contexts the task with the relevant features. Finally, we investigate how the magnitude of contexts affects the performance in task- musk-wwithhng networks.
1 INTRODUCTION
Living involves constantly gauging and selecting the optimal task to perform. This decision is the result of the interaction of different elements, such as our current goals, external circumstances, or the stimulus context (Monsell 2003). In the brain, task switching and cognitive control has been as- sociated principally with the prefrontal cortex (PFC), which provides top-down regulation other cortical brain areas Johnston et al. (2007). In this sense, PFC controls the activation of multiple neural pathways that are activated or deactivated to ultimately result in task execution. Neural path- ways are formed by collections of neurons that cooperate yield specific effect. Although single neurons have been historically regarded as centre of processing in brain, now we are moving to- wards a framework in which neural networks conform the functional processing unit of the nervous system Yuste (2015). In the visual cortex, where subpopulations of neurons responsible of the processing of different visual field features for correct object recognition DiCarlo al. (012))
According to the gating theory (Miller et al. 2001), context-dependent top-down inputs from the PFC regulate the activation of neural pathways. The encoding of relationships between stimulus and contexts can be facilitated by tuning neural activity multiple tasks. In the PFC of monkeys, it has been observed that single neurons exhibit nonlinear responses to multiple stimuli (Rigotti et al.) 2013). This behavior, or mixed selectivity, favors high-dimensional representation of neural activation, which allows linear readouts to generate a vast number of responses, in opposition low-dimensional representations (Fusi et al. 2016).
Recently, artificial neural networks have been revisited as models of neural computation, many findings suggesting their practicality for assessing brain MecoriesRicharde et al. (2019). For example, the internal representations learned by neural networks have been associated with representations the brain in multi-tasking settings (Ito et al. 2022 Ito& & Murray 2021 Flesch el al. 2202a b).
In this paper, we investigate the neural representations learned by feedforward multi-tasking archi- tectures. Neural networks have been designed to be capable of processing multiple tasks in parallel,

---

Under review conference paper ICLR 2023
which beneficial for the network performance (Caruana 1997) to achieve high performance across domains Ruder (017)) Here, we focus on neural networks using contexts to switch attention be- tween tasks. We use population analysis tools to investigate how neural computations are associated (2008) ; Jazayeri & Ostojic (2021) and describe the advantages of learning representations using twit-ssit networks.
with task stimulus Kriegeskorte
1.1 RELATED WORK
Neural networks with contexts have been used as models to study cognitive control in machines and humans. Mante et al. (2013) reproduced PFC dynamics from monkeys using recurrent neural networks with sensory contexts. Similarly, (Ardid & Wang (201)) analyzed task-switching behavior effects, such as switch and congruency, emerging from network attractor dynamics. In Musslick] et (201)) authors studied the learning efficiency of neural networks in multi-tasking with contexts and tasks with ranging degrees of overlapping- |Flesch et al. (2022a) analyzed the geometry of representations learned by neural networks and humans in task-switching schedule. Later, Flesch et (2022b), modified the stochastic gradient descent algorithm to strengthen task-relevant features continual learning setting. More recently, IIto et al. (2022) studied the generalization of new tasks by composing old tasks using different rule-basee contexts. Ito & Murray (202I). authors used a neural network to investigate the tramsformation mapping between visual and motor representations occurring in the brain. They used representational similarity analysis study the geometry of neural codes in multi- tasking Kriegeskorte al. (2008); Kriegeskorte & Kievit (2013).
Contexts have been used to alleviate catastrophic forgetting in continual learning. Masse et al. (2018) showed that adding context to parameter stabilization methods improved accuracy when compared parameter stabilization alone. (Serra al. (2018) implemented gating by using task-based attention mechanisms to help preserving information of previous tasks without compromising learning new
tasks. In Grewal et al. (021). authors combined synaptic intelligence with active dendrites and sparse activations reduce catastrophic forgetting. The idea of recruiting multiple subnetworks for continual learning was previously explored Wortsman et al. (2020).
In addition, Li et al. (2016) proposed multi-bias non-linear activations to improve feature extraction in convolutional neural networks, and for semantic segmentation, task-switching networks using task embeddings were introduced by (Sun et al. (202)) to promote learning common parameters across the tasks using the same network.
1.2 OUR CONTRIBUTIONS
The main contributions of this paper are two-fold:
1. Firstly, investigate the representations Iearned by three different variations of feedfor- ward networks. We find that task-switching network operate in an intermediate regime between individual and parallel networks and the performance on all tasks is improved when parameter sharing is present (section 3.1).
Secondly, we expand previous analyses involving multi-task learning with contexts and mixed selectivity (section 3.3) and report new findings on the impact of contexts location
and magnitude at different stages of processing (sections 3.2 3.4 3.5
2 METHODS
2. ARCHITECTURES
We conducted different experiments with three variants of the feedforward network (Rumelhart
et al. 1986). (Goodfellow et al. 2016) in multi-task learning:
1. Individual Networks: Each task is learned by an independent network. Multi-tasking is
performed by combining the outputs of the networks. An individual network is parameter- ized as ys = fx(x; 0.), where t T denotes the task in the set T (Ta, TR,,,, Tn) ) of N tasks, 6, denotes the weights and biases specific for each task, W. and b, for each layer. Parameters here independent, hence not shared across tasks.

---

Under review conference paper ICLR 2023
Individual
Parallel
Task Switching
TA TB
TB
a
a DO
TA
TB
TA
TB
Figure 1: Architectures for nulti-task learning addressed in this paper.
Parallel Networks: Tasks are processed simultaneously and the output layer accounts for independent responses each the tasks. The parallel processing regularizes the net- work and allows overall better performance than the individual networks Caruana (1997). A parallel network defined y fx(x;0). The parameters in hidden layers actly the same for both tasks, hence fully shared. The only independent parameters in this network are the ones present in the output layer.
Task-switching Networks: Each task processed individually, but parameters, other than the context biases, are fully shared. The output of a task-switching network is = f((;; W, b.). Weights are co-optimized for all tasks, but bias vectors are differ- ent. We define task-switching network to be a composition context bias for each task, y = W + by or not
layers. which can the WTx. A task-switching network requires at least context layer with bias. We train task-switching networks following the interleaved task-switching schedule (Flesch
2018).
Architectures are depicted in figure [ scrihed in section A.
The detailed description of the training conditions are de-
2.2 TASKS
In this study, we created five binary tasks using the MNIST database (LeCun et al. 1998). The main task pair we throughout the study corresponds parity and value. Parity defines a digit either even or odd, whereas value defines digit being smaller equal to four, larger. We analyzed the impact of congruency in learning representations. For two different tasks, stimulus (digit) is congruent if the output labels for both the same. The opposite is referred incongruency. The detailed list tasks outputs is explained section
3 RESULTS AND DISCUSSTON
3.1 SWITCHIIG NETWORKS OPERATE IN AN INTERMEDIATE REGIME BETWEEN INDIVIDUAL AND PARALLEL NETWORKS
The first experiment consisted in analyzing the neural representations of three different multi-task learning networks. Similarly (Caruana 1997), compared multi-tasking using individual neural networks and parallel neural networks. In addition, we included task-switching networks, where tasks are informed by binary context inputs.
To analyze the representations learned by each network, used the representational similarity analysis (RSA) Kriegeskorte et al. (208). We build the representational dissimilarity matrix (RDM) by grouping, in terms of digit images, the neural activations of each layer. For each digit, we averaged the neural activity and computed their dissimilarity. We repeated this process for different tasks and calculated the dissimilarity between class labels (digits). Finally, we assembled RDM. Section A.3) addresses the calculation of the RDM in detail.
For individual and twik-swit networks, calculating the dissimilarity between tasks can be done directly. For individual networks, have specific activity patterns from each network performing

---

Under review conference paper ICLR 2023
a
Layer
Layer
Layer
Layer
Layer
Representational Dissimilarity Matrix (RDM)
Individual
Parry
Diocimbertt
Parallel
Value
Task Switching
Party
Value
1.0
1,000
Peatson Correlation
0.9
0.05
0.900
0.8
Individual Paralial
Apcuracy
0.885
0.7
Individoull TTark Swithin
Paralel: Swithing
0.975
0.5
6470
Instedual
Panslel
Switching
0.900
Party
Value
John
Layer
Task
Figure (a) Average RDM for parity/value task-switching schedule for the three architectures for 10 different runs. (b) Pearson correlation coefficient between RDMs. (c) Accuracy of the differ- ent models.
each task, and for task-switching networks, we have specific activity patterns originated from the same network under each task context. For parallel networks, since the internal processing for both tasks is exactly the same, the resulting neural representations are entangled. Then, we construct the RDM by simply repeating the same activation patterns.
Figure 2 shows the average RDM for the three models after training with parity and value. Individ- ual networks do not share representations between tasks (off offfddiagnaa).. Parallel networks separate partially the congruent and incongruent digits to resolves the active task the output layer. Task- switching networks behave similarly to parallel networks in the first layers but resolves the task before the output layer, similarly the individual networks. Importantly, task goal from the activity of both individual task-switching networks. However, we can only reconstruct the task congruency from parallel networks. Additionally, in comparison to individual networks, task-switching networks share representations between tasks. To quantify the degree of similarity between representations, we calculated the Pearson correlation coefficient between RDMs for different runs. Results in figure 2p reinforce the previous assessment: task-switching networks behave similarly to parallel networks in the first stage of processing, but ends significantly more similar to individual networks.
reconstruct the
Finally, we assessed the performance of networks and analyzed the performance in terms of the accuracy achieved across runs for both tasks. Figure 2 shows similar performance of the three networks for the two tasks, yet significant differences appeared between individual and parallel works, as well as between individual and switching networks, when performance was evaluated simultaneously for the two tasks (Wilcoxon Signed Rank Test, < 0.05). In contrast, parameter sharing between tasks in parallel and task task-switching networks regularizes the network performance.

---

Under review conference paper ICLR 2023
We conducted the same analysis of tigureBusing Dusng different tasks. The results are shown in figure
Y
3.2 TASK-SWITCHING NETWORKS PERFORM MULTIPLE TASKS BY EXTRACTING TASK CONGRUENCY INFORMATION
The second experiment consisted in analyzing the representations learned by task-switching net- works when contexts were removed after training. Tasks share a common pool of neural activations that is biased towards resolving one task or another depending mon computation, first we investigate the RSA by calculating the correlation between RDMs of three variants of the task-switching network: only context in the first layer (first), context in all lay- ers (all), and trained with all contexts but removed at inference time (removed). Figure Bh shows the RSA between these networks. The removed and all networks have similar representations the first layers, only to diverge as we approach the output layer. Contrarily, first and all start with lower correlation but by layer they reach maximum correlation. The drop of correlation in the first layers caused by the interference of the only context with the feature extraction of digits in the first network.
the context. To study this com-
To further amalyze the representations, we computed multidimensional scaling (MDS) on the RDMs (Jazayeri & Ostojic 202I). For implementations details section A.4 Figure Bp shows the MDS for the three first layers and the last two. Digits are represented differently for both tasks in the first network, since the context bias has been introduced early in the processing. Here, the network is forced to resolve the task before the context input dissapears. In layer 3 of the all and removed networks, we see the projection of digits into almost orthogonal dimensions: one dimension represents incongruent stimuli, whereas the other represents congruent stimuli.
In addition to MDS, we used generalized linear model to decode the activations in the original dimensional space (see section (A5)) Figure 31 shows the performance of the classifier when we tested the linearity of digit, task and congruency. high-dimensional space, digit decoded with high accuracy in the first layers. For all three models, the accuracy collapses in the last layers. The removed and all networks are slightly better than first, possibly due to the context interference in the first layer. Task information is already present in the first layer of the first network. The network shows low accuracy in the first layers, as if the contexts where not acting before the third layer (more about weight importance in section B.4). Finally, to test the hypothesis that task switching networks extract congruency information, we evaluated the performance on congruency itself. The removed network is capable of decoding perfectly congruency across layers in the high-dimensional space defined by the neural activations. Here, the first network shows the greatest decrease in accuracy when approaching the last layers. This plot confirms the hypothesis that contexts hias neural activity towards the goal of the task. During training, both tasks are being optimized together, reaching some local minima that reduces the loss for both tasks. Congruency is the midpoint between the optimization of both task and determines the amount shared representations. of two identical tasks, the overlapping is maximum and the bias signals would converge, reducing then the multi tasking problem single-task problem, with contexts working regular activation biases.
the extreme case
3.3 MIXED SELECTIVITY EMERGES FOR RESOLVING TASK EXECUTION
After having inspected the behavior of neural populations in single layers, we wanted to investigate the role of single units, specially in the last layer, where the network needs predict the appropriate output. We hypothesized that neurons have to organize to drive collectively the dynamics towards the task goal. Mixed selectivity has been previously observed in artificial neural networks with a single hidden layer and contexts in the first layer (Flesch et al., (022a). Here, we analyzed the activations of neurons in last layer and performed agglomerative hierarchical to cluster neural activity. The method employed is detailed in section A.6
The results of clustering are shown in figure 4 The different plots represent the mean activity of the different clusters for each task when presented with one of the digits. We used two networks from the previous section, all and removed. The activity was normalized to a maximum of (radius of the plot). We observed the existence of two different clustering patterns. In both, we found a cluster of silent units that is not active for any of the numbers. The first row of the plot represents

---

Under review conference paper ICLR 2023
a
Removed
All
Pearsbn Correjation
0.9
First
All
0.0
Removedd
All
0.5
First All
0.4
Removed
Layer
b
Layer 1
Layer 2
Layer 3
Layer 9
Layer 10
16
First
NOTZ
NO2
NOZE
All
NOCZ
MOGT
Removed
MESII
MODS
Partt
Vilue
Number
Congruency
10-
1.0
1.0
Acturary
0.6
0.6
8.
Compotiial
0.4-
04
0.0
10
Lager
Layer
Layer
Figure 3: (a) Mean and standard deviation of RSA between first, all and removed network for 10 different runs. (b) Multidimensional scaling (MDS) of the representational dissimilarity matrix (RDM). (c) Mean accuracy and standard deviation of the linear decoder of neural activity for 10 different runs.
the first pattern. In this pattern of activation, there is single cluster that responds to even digits in parity and values less than or equal to four in the value task. When contexts are removed the cluster responded only to the even digits and values less than equal to four, entailing that context increases the activity For the other digits, the network thresholds neural activity. For example, odd digits in the parity task show silent activity.
and parity and and value, that is, the incongruent digits.
The second pattern of activation is showed in the second row of figure We found main clusters of activity. The first cluster accounts for even digits and values less than or equal to four, the second cluster for odd digits and values greater than four, When contexts were removed, clusters only encoded the congruent digits. These patterns of activations show how neurons tundd

---

Under review conference paper ICLR 2023
Parity
Value
Removed
Pattern 1
2
Pattem 2
8
X
Neural
Nousal
Figure 4: Neural clustering for the parity/value task-switching schedule.
different depending on current active stimulus. the brain, neurons that are responsive multiple tasks are said to exhibit mixed selectivity (Rigotti et al. (013)). We repeated the experiment using five tasks and found that the majority of patterns resembled pattern number two. These results appear figure 7 the appendix.
3.4 CONTEXT STRENGTH IS MAXIMIZED AT INTERMEDIATE LAYERS
After observing the impact of context in the first layer in the previous section, we hypothesized that context location could be essential for proper input feature extraction and to drive the activity towards task goal. Flesch et al. (202aa) observed anticorrelation in the contexts tasks situated in the first layer in networks with one hidden layer. Here, we run a series of experiments to assess the strength of contexts varying their position across the network. Figure shows the results. Each row represents different context configuration and the maximum number of layers to 10). A cell colored according to the heatmap means that the network has context that position. A gray cell means that the network has layer but context. white denotes the absence of layer, and thus network shallower. We tried four different settings: adding contexts from the first layer (5)), adding contexts starting from the last layer (5)). adding single context at different layers (5E) and adding contexts and layers the same time (āi).
column is index of the layer (we fixed
two first heatmaps we show the Frobenius norm of the weights associated with the corresponding task. Then we calculated the correlation between contexts of different tasks at the location. Finally, show the average accuracy (and standard error) of the networks in the same row, The dashed line the accuracy plot represents mean accuracy achieved by network of 10 layers and 10 contexts (control case).
Results show that context location is not trivial. Largest context strength is achieved at intermediate layers, where the norm of contexts is higher. At the time, the anticorrelation between contexts of different tasks is higher. Context have positive correlation if located at the end networks with multiple contexts. In addition, adding context at the beginning at the end of networks are asso- ciated with the largest magnitude of context strength. It is explained by the fact that the network constrained to digit processing and task separation the former case, and task scparation and output mapping in the latter. When assessing the accuracy, the presence of contexts in intermediate layers achieved the overall performance to that of the control Finally, compared the biases

---

Under review conference paper ICLR 2023
a
Parity
Value
Correliaion
Accuracy
Netwolkk Index
00
Network lidex
100
10
000
Network Indeex
0.0
-1.0
Network Index
5.0
0.0
25
0.0
-1.0
Layer Index
Layer Index
Layer Index
Figure 5: Mean magnitude and correlation between contexts using different network architectures for 10 different (a) Context are added from the first layer. (b) Context added from the last layer. (c) context per network. (d) Networks have different number layers.
between individual and lask-switching in figure y and show that large magnitude intermediate layers is exclusive for task-switching networks.
3.5 TASK-SWITCHING NETWORKS GENERALIZE REPRESENTATIONS BETWEEN TASKS
The last analysis involved the study of the neural representations when the magnitude of contexts was modified from the values of training. For each task, since we binary (one-hot) encoding, the contexts have a default value of if the task is to be executed and 0 otherwise. Values between 0 and can be for instance related to task uncertainty, either normalized to sum I or independently from one another. Hence, many contexts could be activated at the same time. We used a 10 layer network with contexts in all layers. In a last experiment, we analyze the behavior of the task-switching network against abnormal context values (>1), which interest is to analyze task performance under aberrant context modulations. such those that could be originated from abnormal activity PFC. these scenarios, we inspect the neural representations to gain insight how networks perform tasks.
First, studied the how is the transition from task another. To do values of contexts between the two tasks, passing by an intermediate value of 0.5. The total sum the contexts was kept to I, and values are complimentary. As mentioned carlier, this could be related task uncertainty in normalized fashion. Figure 6 shows the accuracy for the different values of the context. moved from task value to task parity. Results show that the transition between tasks is smooth: there is not a hard threshold at 0.5, but progressive change from one task to the other. The neural representations 0.5 show intermediate state between tasks. The baseline accuracy 0.6 since is the level congruency between tasks. The accuracy at 0.5 close 0.8, which the random chance once the congruency baseline considered.
interpolated the
Next, we wanted the strength of opposition between tasks. We incremented the value of both con- simultaneously from 0 to This way parametrically analize performance according

---

Under review conference paper ICLR 2023
a
b
64-
0.-
Accuracy
Acuurayy
Partty
0.7
Volue
07-
0.0
02
044
0.6
0
1.0
00
02
0.4
0.6
0d
3.0
C
Context Strongth
d
Context Strength
Accuracy
0.5
Accursyy
07
0s-
0.-
84
2/4
Strength
Strengtt
Figure 6: Mean accuracy of tasks as functions of context intensities for 10 different runs. Shaded area denotes standard deviation. (a) The total context strength is kept fixed between tasks. (b) Both tasks are increased simultaneously. (c. d) Contexts are increased and evaluated independently (no interaction).
context strength when both tasks are equally probable. Figure 6p shows the accuracy values for both tasks. The accuracy begins at 0.8 and slightly diverges when reaching value of 1 for each task, which is a slight preference of the networks for the task parity. The internal representations however, show that the network operates in intermediate step between tasks as observed in figure Gb when both tasks had contexts 0.5.
Finally, we wanted to analyze how the network performed each single task separately when the values of the contexts were less greater than one. As mentioned earlier, the former can be related non-normalized task uncertainty. whereas the latter is useful to infer behavior against aberrant context inputs. In figure bg. we swept each task independently from 0 to I, and represented the values in the same plot. Results indicate that networks do not need to fully certain of the magnitude context to operate at highest performance. Concretely, we observe that with contexts close 0.5, the networks reach the maximum accuracy. We took each network and created the RDM. The results show that the neural representations for binary (full certain) contexts for uncertain contexts if their strength is greater than 0.5. Figure 6 shows the accuracies for the independent tasks when the values of contexts are up to 30-fold of those in training. When the context dramatically increases, there is asymmetry between digit extraction and task information that leads decrease accuracy. is like the network very clear about the task perform but has lost most of the information about the digit. Hence, the dissimilarity hetween mean activations becomes close to zero.
already present
4 CONCLUSIONS
In this study, have conducted series of experiments with multi-task networks primarily focus- ing on task aask-switching networks. which leverage contexts to switch between tasks. We have seen that task-switching networks learn disentangled shared representations that allow to reconstruct the map- ping between task and goal. Task-switching networks encode congruency to resolve multi-tasking, Contexts can either improve feature extraction or interfere with the network processing, depending on their position in the network. Finally, we have shown that fask-switching networks generalize representations between tasks.

---

Under review conference paper ICLR 2023
REPRODUCIBILITY STATEMENT
We attached int ZIP file the code train and generate the figures together with the submission of this paper.
REFERENCES
Salva Ardid and Xiao-Jing Wang. A tweaking principle for executive control: neuronal circuit mechanism for rule-based task switching and conflict resolution. Journal of Neuroscience, 33
(50):19504-19517, 2013.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
James DiCarlo, Davide Zoccolan, and Nicole Rust. How does the brain solve visual object recognition? Neuron, 73(3):415-433 2012.
Timo Flesch, Jan Balaguer, Ronald Dekker. Hamed Nili, and Christopher Summerfield. Comparing continual task learning in minds and machines. Proceedings of the National Academy of Sciences, 115(44): 2018.
Timo Flesch, Keno Juechems, Tsvetomira Dumbalska, Andrew Saxe, and Christopher Summerfield. Orthogonal representations for robust context-dependent task performance in brains and neural networks. Neuron, 10(())1258-112) 2022a.
Timo Flesch, David G Nagy, Andrew Saxe, Christopher Summerfield. Modelling continual learning in humans with hebbian context gating and exponentially decaying task signals. arXiv 2022b.
preprint 2rX3v:2203.11/33
Stefano Fusi, Earl K Miller, and Maltia Rigotti. Why neurons mix: high dimensionality for higher cognition. Current opinion in neurobiology, 37:66-74, 2016.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 315-323. JMLR Workshop and Conference Proceedings, 2011.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume MIT Press, 2016.
Karan Grewal, Jeremy Forest, Benjamin P Cohen, and Subutai Ahmad. Going beyond the point neuron: Active dendrites and sparse representations for continual learning. bioRxiv, 2021.
Takuya Ito and John D Murray. Multi-task representations in human cortex transform along sensory-to-motor hierarchy. bioRxiv, 2021.
Takuya Ito, Tim Klinger, Douglas H Schultz, John Murray, Michael W Cole, and Mattia Rig- otti. Compositional generalization through abstract representations in human and artificial neural networks. arXiv preprint arXiv:2209.04331, 2022.
Mehrdad Jazayeri and Srdjan Ostojic. Interpreting neural computations by examining intrinsic and embedding dimensionality of neural activity. Current opinion in neurobiology, 70:113-120, 2021.
Kevin Johnston, Helen M Levin, Michael I Koval, and Stefan Everling. Top-down control-signal dynamics in anterior cingulate and prefrontal cortex neurons following task switching. Neuron, 53(3):453-462, 2007.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1VXx.::412.6980) 2014.
Nikolaus Kriegeskorte and Rogier A Kievit. Representational geometry: integrating cognition, com- putation, and the brain. Trends in cognitive sciences, 4718):001-41, 2013.
Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis- connecting the branches of systems neuroscience. Frontiery in systems newroscience, pp. 4, 2008
10

---

Under review conference paper ICLR 2023
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied document recognition. Proceedings of the IEEE, 86(1 (0:2278) 2324,, 1998.
Hongyang Li, Wanli Ouyang, and Xiaogang Wang. Multi-bias non-linear activation deep neural networks. In International conference on machine learning, pp. 221-229. PMIR, 2016.
Valerio Mante, David Sussillo, Krishma V Shenoy, and William T Newsome. Context-dependent computation by recurrent dynamics in prefrontal cortex. nature, 503(7474):78 84, 2013.
Nicolas Y Masse, Gregory D Grant, and David Freedman. Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization. Proceedings of the National Academy of Sciences, 115(44)
2018.
Earl K Miller, Jonathan D Cohen, et al. An integrative theory of prefrontal cortex function. Annual review of neuroscience, 24(1):167-20) 2001.
Stephen Monsell. Task switching. Trends in cognitive sciences, 7(3):134-140, 2003.
Sebastian Musslick, Andrew Saxe. Kayhan Özcimder, Biswadip Dey. Greg Henselman, and Jonathan D Cohen. Multitasking capability versus learning efficiency in neural network archi- tectures. Cogmitive Science Society, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, 32, 2019.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and Duchesnay. Scikit-learn: Machine learning Python. Journal of Machine Learning Research, 12:2825-288 2830, 2011.
Blake Richards, Timothy Lillicrap. Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. A deep learning framework for neuroscience. Nature neuroscience, 22(11): 12(11)176111170, 2019.
Mattia Rigotti, Omri Barak, Melissa R Warden, Xiao-Jing Wang, Nathaniel D Daw, Earl K Miller, and Stefano Fusi. The importance of mixed selectivity in complex cognitive tasks. Nature, 497 (7451):585-590, 2013.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv: 05098, 2017.
David Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by back- propagating errors. nature, 323(6888) 1986.
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In International Conference on Machine Learning, pp. 4548-4557 PMLR, 2018
Guolei Sun, Thomas Probst, Danda Pani Paudel, Nikola Popović, Menelaos Kanakis, Jagruti Patel, Dengxin Dai, and Lue Van Gool. Task switching network for multi-task learning, In Proceedings of the IEFF/CV International Conference on Computer Vision, pp. 8291 2021.
Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Coumaaeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. Scipy 1.0: funda- mental algorithms for scientific computing in python. Nature methods, 17(3)2611-277 2020.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. Advances in Neural Information Processing Systems, 33:15173-15184 2020.
Rafael Yuste. From the neuron doctrine to neural networks. Nature reviews neuroscience, 16(8): 487-497, 2015.

---

Under review conference paper ICLR 2023
A APPENDIX
A.1 NETWORK TRAINING
For each architecture, we trained two different models with 3 (section 3.1 and 10 layers (sections
3.2
B.5). Networks were trained for total number of 50 epochs. The number of hidden units per layer fixed to 100. We selected ReLU the activation function et (201 I) and initialized weights and biases following uniform distribution the range (-//x, M/N), where N is the number of inputs to the layer. We used the Adam optimizer for training (Kingma & Ba 0.9 and 82 0.999 and the
3.4
2014) with learning rate 0.001,
entropy loss function. The
models were implemented in PyTorch Paszke et al. (2019). We collected statistics over 10 different runs and controlled the random number generator for reproducibility and hypothesis testing. The code to reproduce the results is attached in ZIP file with the submission of this paper.
A.2 BINARY TASKS
We designed five main tasks using MNIST, For all tasks, we split the images into training set of 50,000 images and test of 10,000. The batch size was fixed 100 images. The following table denotes the names of the task and the expected output of the network (0, first output unit, second output unit):
Number
Task Name
0
1
2
5
6
7
8
Parity
1
0
0
0
1
0
1
0
Value
1
1
0
0
0
0
Prime
0
0
0
1
0
Fibonacci
1
1
0
1
0
0
0
Multiple 3
0
0
0
0
0
The total congruency between tasks is represented in the following table:
Parity
Value
Prime
Fibonacci
Multiple 3
Parity
1
0.6
0.3
0.5
0.4
Value
0.6
1
0.5
0.7
0.4
Prime
0.3
0.5
0.6
0.5
Fibonacci
0.5
0.7
0.6
0.3
Multiple 3
0.4
0.4
0.5
0.3
A.3 REPRESENTATIONAL DISSIMILARITY MATRIX (RDM)
For each layer, construct an activation matrix consisting of NT x Niest rows and Nhidden columns, where .r is the number of tasks, Nrest is the test set size, 10,000 and Nuidden the number of hidden units a layer. For each label of MNIST, we will calculate the average activation for the hidden units in the layer. For each task, we create matrix of dimension 10 x Nasaden- This matrix represent the average unit response each of the possible stimulus of the dataset. Finally, we calculate the dissimilarity between row vectors, denoting the dissimilarity between the average of two numbers for two different tasks. We calculate the dissimilarity between vectors as 1 f/2, where is the Pearson correlation coefficient.
A.4 MULTIDIMENSIONAL SCALING (MDS)
To construct the multidimensional scaling embedding we create first the RDM of layer. The RDM distance matrix where position ij denotes the distance between and j. We project this high- dimensional points into a two-dimensional embedding using MDS. We used the scikit-learn module to implement MDS Pedregosa et al. (201I). The maximum iteration was set to 1000 to guarantee convergence. The tolerance
12

---

Under review conference paper ICLR 2023
A.5 LINEAR DECODER
We used multiclass logistic regression as linear decoder neural activations across layers. We used the scikit-learn module to implement the logistic regression model Pedregosa et al. (201I). The maximum iteration used L2 regularization and cross entropy as loss function.
5000 guarantee convergence. The tolerance
10 We
We constructed an activation matrix consisting of Nr X Neett rows and Nasdden columns, where Ny is the number of tasks, Nest is the test set size, 10,000 and Nhidde the number of hidden units train set and test set following a 90/10 split. We
of layer to decode. We divided the rows used the labels of numbers to create the task and congruency labels. We assembled the activation matrix for each layer of network and averaged the results of the 10 different runs.
A.6 AGGLOMERATIVE HIERARCHICAL CLUSTERING
Similarly as in the assembling of the RDM, we average the neural activity for the 10 numbers, having matrix of size 10 Nandden- Here gather the averages of activity for the different tasks construct matrix of size NAudden x 10 Ny. Each independent row conforms the averages activations across tasks for a single hidden unit. We normalize the activity to be maximum of 1. We apply hierarchical clustering to create the dendrogram of hidden units using the centroid method in SciPy (Virtanen al., 2020). The we find the cluster of each hidden unit. Finally, we calculate the average activity of each cluster.
13

---

Under review conference paper ICLR 2023
SUPPLEMENTARY FIGURES
a
Layer
Layer 2
Layer
Layer
Layer 5
Individual
Parallel
Task Switching
b
C
1.0
1.00
0.99
Pearson Correlation
0.96
Accuracy
0.6
Inchvidusl Poraiel
Inciviiual Switching
0.87
Paralii Trac Swithing
0.56
Indiviuull
0.55
Partlell
0.2
Swithing
0.94
Party
Vodee
Priee
Reooaccl husples 3
2014
Layer
Task
Figure 7: (a) Average RDM for puttsyeluepiimm//hooccchmlaagp task- switching schedule for the three architectures for 10 different runs. (b) Pearson correlation coefficient between RDM (mean of 10 runs with standard deviation). (c) Accuracy of the different models.

---

Under review conference paper ICLR 2023
Parity
Value
Fibonacci
5
5
Prime
Multiple 3
Removed
0
Neural Cluster
Cluster
Figure 8: Neural clustering for the partthyuaheehhrrmmfffionacimmnem wtsp:///wwmmiiiieeeessssstiiiic task-switching schedule.
a
1.00
Parry S00ue
Woightt Magnitudo
0.50
Corrolation
0.25
6.00
-0.25
-0.50
-1.00
10
Index Layer
Index Layer
0.75
Weight Magnitude
0.50
Corrolation
0.25
3--
0.00
2-
-0.75
Index Layer
Index Layer
Figure 9: Magnitude and correlation hetween hiases in individual (a) and task-switching networks (b). We show mean and standard deviation over runs.
15


Reviews:
[{"review_id": "w8_TV3BsF4", "reviewer": "Reviewer_opgD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper compares three network architectures for multi-task learning. The experiment uses the MNIST dataset for two binary classification tasks. The authors categorized the output activation of each layer according to their input number to show their (un)correlation with the task. The authors also analyzed the structure of the task-switching network by changing the additional method or location, order, or strength of context vectors. Based on the results of these experiment and various analyzing methods, they present special features of task-switching networks. This paper proposes that adding context helps the network to generate expressive features and map tasks to appropriate outputs.", "summary_of_the_review": "The authors did many experiments. But experimental results and their findings is not strongly connected.\nThe tasks they defined are highly correlated, so it is not possible to be sure that the results came from the context or the task, which is insufficient to supporting their findings.\nToo many typos make uncomfortable experience.\n\n", "strength_and_weaknesses": "Strengths\n- Various analysis technique\n- Well visualized results\n\nWeaknesses\n- Miss guided figure numbers make readers confusing. Correction is must needed, especially in the first half.\n- I think tasks are too easy that switching-task might be not meaningful. As you showed in Figure 2(c), all models work well in both tasks. So Figure 2(a) has no meaning because outputs are too obvious. If you want to claim parameter sharing of networks, tasks should not be correlated, like the number of input and average value of input pixels.\n- Why did you use one-hot vector as context vector? Adding not trainable one-hot vector as bias makes 'biased' networks. I want to see the activation values and its means. Dose adding 1 make too big changes in its output? If it is, the experiments about the context strengths is just come from the added bias, not the context. Using embedding vectors is reasonable choice if you care about the gap between value 1 and activation.\n- In figure 5, you should show magnitude of individual network trained with one-hot vector used as bias to prove magnitude difference come from context.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "i8FH1ClGmS", "reviewer": "Reviewer_Qimc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates neural network representations under multi-task learning based on five binary classifications tasks constructed from MNIST.  The authors compare using independent, individual multi-layer perceptrons (MLPs) per task with parallel MLPs that process tasks simultaneously and with task-switching MLPs that share weights between the individually processed tasks. They show that task-switching networks learn disentangled shared representations that generalize between tasks and encode congruency.", "summary_of_the_review": "The work is interesting but the limitations in choice of networks and datasets make it not strong enough for ICLR. Follow-up work that addresses the limitations will help to qualify the paper,", "strength_and_weaknesses": "The work investigates an important domain, is improving the understanding of neural network representations under multi-task learning may be helpful for improving multi-task learning and explaining model predictions. It remains however unconvincing as 1) the authors do not verify if their results hold across multiple runs with different random initializations of the networks or are only random artifacts 2) MNIST is a narrow and small dataset, findings based which may not transfer to state-of-the-art applications of neural networks for example in computer vision or natural language processing which were found to benefit from multi-task learning.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MvXSF5Tg4e8", "reviewer": "Reviewer_W7yc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of neural representation learning in multi-task learning. The authors compared three types of model architectures: individual network, parallel network and task switching network, on MNIST dataset with a multitask learning setting. To be more specific, the authors conducted analyses on the representations learned from these model architectures. In the discussion, the authors pointed out that task-switching networks can encode congruency to resolve multi-tasking, and the context of tasks can improve multi-task learning differently when being used at different positions (layer depth).", "summary_of_the_review": "This paper conducted experiments and analyses on understanding neural representations in multi-task learning networks. There are some very interesting results and discussions. However, I think this paper can be further improved by including more recent advances in multi-task deep learning and discussion related to their findings. Meanwhile, the experiment section can be improved by including larger benchmark datasets.", "strength_and_weaknesses": "Strengths\n- This paper studies a very meaningful research problem of understanding the representation learned for multi-task models. \n- The authors utilized different statistical tools to conduct analyses on understanding the representation learned and importance of context for different multi-task learning model architectures.\n\nWeaknesses\n- Presentation. The presentation and organization of this paper can be improved. I found it hard to connect different analyses done in section 3. It might be good to have an overview of different research hypotheses or questions being answered in each of the analyses.\n \n- Novelty. The technical novelty of this paper is relatively limited. The model architectures used in this paper to conduct analysis don’t reflect recent advances in multi-task deep learning. Also the analyses being done here is similar to other existing work that studies the understanding of multi-task deep learning. I also want to see if the authors can propose some novel improved techniques based on their findings. \n\n- Related work. This paper doesn’t include recent advances in multi-task deep learning, either in understanding MTL or MTL model architectures, which limits its contribution and significance. For example, [1] conduct analysis on multi-task deep learning for its generalization ability related to representation learning and model capacity. Also, besides the task switching network, Mixture-of-experts [2,3] based approaches, and attention modules [4] have been widely used for multi-task learning.\n\n[1] https://arxiv.org/abs/2005.00944\n[2] https://proceedings.neurips.cc/paper/2021/hash/f5ac21cd0ef1b88e9848571aeb53551a-Abstract.html\n[3] https://dl.acm.org/doi/pdf/10.1145/3219819.3220007\n[4] https://arxiv.org/abs/1910.10683\n\n- Experiment dataset. The dataset used in conducting the experiment is quite small. Multi-task learning generalization benefits can be better evaluated in different scenarios, such as: (1) tasks have similar amounts of data, (2) some tasks have much fewer data (3) few-shot tasks. These scenarios cannot be fully captured by MNIST dataset.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KtWP593XlTB", "reviewer": "Reviewer_px1r", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates the effect of task-dependent context in multi-task learning for the task-switching networks. Particularly, the authors show the task-switching networks operate in a regime between two extreme cases of individual and parallel networks. The authors also study the shared representations, the role of context and its magnitude in simple multi-task learning experimental settings. ", "summary_of_the_review": "Overall, I found the exploratory analysis of context dependent multi-task learning an important analysis to be done. However, I believe the paper needs some work to either show the results hold in more general settings and difficult tasks, or provides some guideline on how to utilize the insights for improving multi-task learning algorithms. ", "strength_and_weaknesses": "The paper studies an important problem which I believe it is not well studied yet. However, I have the following questions and comments for improving the paper:\n- For the task switching network, having separate bias terms for each task makes sense but might be too simplistic. It would be interesting to do similar studies to what you already have with more flexible transformations such as the one proposed in Sun et al., 2021. \n- minor: colorbar in Figure 2(a) seems to be incorrect (all red).\n- Figure 3.1b in the text seems to refer to Figure 2.\n- “switching networks, behave similarly as parallel in the first tasks” it should be in the first “layers”. Also the section is missing in “After observing the impact of context in the first layer (section ), “. Generally, the paper can benefit from some careful proofreading. \n- The title for section 3.4 might be misleading as Figure 5.C doesn’t support that. Also I’m surprised that having a single layer with context is doing so well in terms of accuracy. This may be a hint that the task is too simple for an interesting analysis. A dataset with more diverse images may help in this experiment. \n- How does the analysis change if done on a more challenging dataset like CIFAR-10 and also with a slightly more complicated network architecture like a CNN. \n- I found some results to be interesting but it’s unclear how they can be used for designing better context-dependent multi-task learning algorithms. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject