Mid-Vision Feedback
Authors: Michael Maynord
Eadom T Dessalene
Cornelia Fermuller
Yiannis Aloimonos
Venue: ICLR 2023 poster (2023)

Abstract: Feedback plays a prominent role in biological vision, where perception is modulated based on agents' evolving expectations and world model. We introduce a novel mechanism which modulates perception based on high level categorical expectations: Mid-Vision Feedback (MVF). MVF associates high level contexts with linear transformations. When a context is "expected" its associated linear transformation is applied over feature vectors in a mid level of a network. The result is that mid-level network representations are biased towards conformance with high level expectations, improving overall accuracy and contextual consistency. Additionally, during training mid-level feature vectors are biased through introduction of a loss term which increases the distance between feature vectors associated with different contexts. MVF is agnostic as to the source of contextual expectations, and can serve as a mechanism for top down integration of symbolic systems with deep vision architectures. We show the superior performance of MVF to post-hoc filtering for incorporation of contextual knowledge, and show superior performance of configurations using predicted context (when no context is known a priori) over configurations with no context awareness.

Published as a conference paper at ICLR 2023
FEEDBACK
Michael Maynord, Eadom Dessalene, Cornelia Fermüller, and Yiannis Aloimonos Department of Computer Science University of Maryland, College Park
College Park, MD 20742, USA
[maynord, edessale, fermulcm, jyaloimo} @umdd edu
ABSTRACT
Feedback plays a prominent role in biological vision, where perception is mod- ulated based on agents' evolving expectations and world model. We introduce a novel mechanism which modulates perception based on high level categorical expectations: Mid- Vision Feedback (MVF). MVF associates high level contexts with linear transformations. When a context is "expected" its associated linear transformation is applied over feature vectors in mid level of network. The result is that mid-level network representations are biased towards conformance with high level expectations, improving overall accuracy and contextual consis- tency. Additionally, during training mid-level feature vectors are biased through introduction of loss term which increases the distance between feature vectors associated with different contexts. MVF is agnostic as to the source of contextual expectations, and can serve as a mechanism for top down integration of symbolic systems with deep vision architectures. We show the superior performance of MVF to post-hoe filtering for incorporation of contextual knowledge, and show superior performance of configurations using predicted context (when no context is known a priori) over configurations with no context awareness.
INTRODUCTION
In most contemporary computer vision architectures information flows in a single direction: from low-level of pixels up to high level abstract concepts (e.g.. object categories)- such architectures are termed forward architectures. In general, each successive layer of the network contains more abstract representations than the previous, and the representational hierarchy mirrors the architec- tural hierarchy. It is also possible to introduce top-down connections into the network architecture, introducing high level information into processes involving lower levels of abstraction in a process of feedback.
Feedback plays primary role in biological vision; in fact, the majority of neural connections in the visual cortex are top-down, rather than bottom-up, connections (Markov et al.) 2014). These top- down connections are thought to convey information of higher level expectation, and neurons of the visual cortex use both higher level expectation as well as lower level visual information in producing their representations. Expectations in biological systems arise from continuous engagement with the environment. In Computer Vision, this reflected the paradigm of Active Vision (Bajcsy 1988 Fermüller & Aloimonos 1995), where perception is framed as active problem involving evolving world models.
The task of producing mid-level visual representations Teo et al. (2015a by: Xu et al. 2012); Nishi-
gaki et al. 2012) from low level input is under-constrained many plausible mid-level interpretations may be consistent with input. To give an intuition for how understanding of context can impact per- ception of mid-level features consider Figure [ characteristics of shrews and kiwi differ, but may be similar enough to be confused without context. Top-down feedback from high level context to mid-level visual features provides "map" for mid-level processes, constraining it towards high level consistency.
Coode will he available http:///hhhhii..oommmmmmoodddddd

---

Published as a conference paper at ICLR 2023
Figure 1: Here we illustrate images cropped to exclude context. first glance, due to similarities in color, texture, and pattern, images from the top row (animate) may appear to be of the same class as those of the bottom row (inanimate). With an understanding of the difference of context, upon closer inspection, it is clear that there are meaningful lower level feature differences.
Introduction of contextual knowledge through feedback is superior to post-hoc application of contex- tual knowledge, e.g. through discarding interpretations (classifications, here) which are not context consistent. We demonstrate this point empirically.
Interpretations selected after post-hoc filtering for context consistency will still be built upon under- constrained mid-level features. Furthermore, in contrast to post-hoc filtering, feedback naturally allows for detection out-of-context objects, as feedback functions through biasing visual rep- resentations rather than filtering. It is valuable for methods to allow for out-of-context detections, even when biasing against them, as out-of-context objects on occasion appear (c.g., tree in an office setting).
CNNs have a natural tendency towards decou- pled representations: representations with tendency for feature vector angle to correspond characteristic type (e.g.. "fuzzy"), and for
Veristion across lype
feature vector magnitude to correspond to char- acteristic variation or degree (Liu al., 2018) (e.g., "very fuzzy" "not fuzzy") (See Figure 2 for an illustration). This opens up a couple of possibilities in terms of directly manipulating feature representations:
Variatoo
Origin
over Degree
We can differenti- ate between axes with different associations to high level contexts, 2) we can control magni- tudes of characteristies through amplifying and dampening axes associated with those charac- teristics. That is, w.r.t. point #1, CNNs produce representations which are, to a degree, separated by angle, certain axes will be more associated with some higher level contexts over others. Also, w.r.t. point #2, amplifying char- acteristics associated with a higher level context increases the likelihood of interpreting input as conforming to that context; dampening charac- teristics associated with that context reduces the likelihood of interpreting input conforming to that context.
Artist illustration of decoupled repre- sentations, as may be learned by a CNN. For illus- tration we annotate characteristics with visually recognizable categories, though this work we exploit the tendency towards decoupled represen- tations at lower levels. Feature vector angle cor- responds to characteristic type, while feature vec- tor magnitude corresponds to within characteristic variation or degree. (Liu et al. 2018) observe that CNNs produce decoupled representations, and de- rive similar illustration MNIST by setting convolution operator's dimension to 2.
Figure 2:
We present principled method to feedback Hid-Vision Feedback (MVF), illustrated in Figure - allowing the biasing of mid-level feature representations in networks such CNNs to- wards conformance with high level categorical expectations. This approach is comprised of two components: 1) linear transforms (affine transformations), and 2) orthogonalization bias.
2

---

Published as a conference paper at ICLR 2023
Affine transformations enable direct control over the feature vectors at the injection level the level into which feedback is being inserted. If these vectors have been trained with a bias towards orthog- onality w.r.t contexts, then this allows for affine transformations to manipulate features associated with context presence or absence with less impact on other features.
The orthogonalization bias is introduced to increase the independence between contexts, they can be manipulated with less interference. This bias is introduced at the injection level. This does not negatively impact the representational power performance of the base network. This orthogonality bias introduced across contexts e.g.. mid-level feature vectors associated with animate contexts can be biased towards orthogonality w.r.t mid-level features associated with inanimate contexts. Due to the resulting greater angular separation between features associated with different contexts, this biasing allows greater control over facets of mid-level representation which are meaningful to higher level contexts.
MVF then functions as follows. During runtime high level context expectation associated with input. This expectation is used in biasing mid-level visual features through use of affine trans- formation associated with the context of that expectation. This selects for characteristics associated with this context. These affine transformations are better enabled consequence of the disentan- glement of such characteristics at the injection level, effected through introduction of the orthogo- nalization bias during training.
Feedback then enables a synergy between high level categorical interpretations and mid-level visual feature representations, bridging the signal-symbol gap in both directions. This approach incorpo- ration of context expectations is controlled. This differs from an approach of connecting the upper level fully connected layers of network directly lower level convolutional levels in scheme which includes neither categorical representations nor biasing w.r.t. said categorical representations.
Sample #1
COMV
A
CONV
Horse
Lo
E
Sample
CONV
A
CONV
Table
Figure 3: MVF structure. As training involves a contrastive loss (Lo) between samples belonging different contexts, the base network shown twice in the figure: the top when fed with horse image, and on the bottom when fed with a desk image. Affine transformations (lincar transfor- mations) inserted into a conventional CNN architecture. These affine transformations modulate feature representations at injection levels (network levels over which affine transformations are ap- plied) in accordance with high level context expectations (E). Lo is inserted put pressure on the network to help separate feature representations at the injection levels according to context, better enabling manipulation by affine transformation. During training, samples from different contexts are fed through the network in pairs. The context expectations determine the selection of appropri- ate affine transformations.
MVF both employs feedback from categorical knowledge and is agnostic w.r.t. the source of that categorical knowledge i.c., it is not requirement that context expectations be produced from the same network. As consequence of this, MVF allows for interfacing with larger symbolic systems e.g., models of scenes employing graphical models over scene elements and categories. This top- down synergy across the signal-symbol gap opens up wide range of applications.
The rest of this paper is structured as follows: In Section 2 we cover related work; in Section we detail methods; in Section we cover experiments; and, in Section 5 we conclude.

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
2.1 BIOLOGY, FEEDBACK, AND PARALLELS TO COMPUTER VISION
Previous works (Markov al. 2014), (Gilbert & Sigman 2007), Kreiman & Serre (200)) Gilbert
& Lij 2013), and (Paneri & Gregoriou 2017)) have explored the importance of feedback connec- tions in biological sensory perception. Further work ((Liao & Poggio 2016) and (van Bergen & Kriegeskorte 2020j) draw connections between feedback in computer vision architectures and the primate visual cortex. (Tang et al. 2018) show that feedforward CNNs are not robust to occlusion, unlike in human perception, but that adding recurrence improves occlusion robustness. (Lotter et al., 2016) introduce PredNet, a network based on predictive coding, and demonstrate benefits on the task of self-supervised frame prediction.
There is good reason to believe that modeling characteristies of biological vision computer vision architectures will benefit computer vision (c.g., (Medathati et al. 2016 Teo et al. 2015c)). For example, (Linsley et al. 20206) demonstrate a network with top-down connections which aligns with human perception of visual illusions, where feedback aids in prioritizing object boundary tours over simple edge contours. (Linsley et al.) (220a) shows how recurrent hierarchical feedback model can improve segmentation. (Konkle & Alvarez 2020 introduce instance-prone ppottttye con- trastive learning, and show that self-supervised models can learn representations which are more brain-like than supervised models. (Li al. 202T) introduce Contrastive Clustering, showing a benefit to instance- as well as cluster-level contrastive loss in clustering.
(Long et al., 2018) demonstrate large scale organization of the cortex based on mid-level visual fea- tures (below the level of object recognition). including those associated with animacy vs. inanimacy. (Jgadeess & Gardner 2022) argue that representations in category selective regions of the visual cortex encode a basis representation for texture, rather than objecthood representations. (Harrington Deza 2021) demonstrate that constraining networks to be robust to adversarial input produces network representations more in-line with human visual perception, and argue for the use of texture summary statistic representations.
2.2 FEEDBACK IN EXISTING COMPUTER VISION METHODS
The conventional approach to feedback in computer vision is the use of recurrent connections. (assell al. 2016) introduce recurrent connections into shallow CNN architecture for image classification. (Pinheiro & Collobert 2014) employ recurrency over convolutions for the purpose of enabling lateral information flow in the task of segmentation. (Zamir et al.] 2016) instantiate feedback through an RNN architecture which iteratively refines prediction categories from coarse to specific.
Alternatives to conventional recurrent connections for feedback include (Hu & Ramanan 2016 which explore convolutions with hierarchical rectified Gaussians to enable top-down as well as bottom-up information flow, and apply them to the task of keypoint localization under occlusion. Additionally, (Yao et al. 2012) apply graphical model over scene representations, allowing higher and lower level decisions to influence each other.
3 METHODS
With MVF we seek feedback mechanism which allows us to directly bias lower level feature repre- sentations based on categorical higher level context expectations. This involves top-down interaction across two levels of abstraction: 1) high level contexts 6 C, 2) mid-level features f, E F.
The structure of MVF is illustrated in Figure and the loss and training formulation given in Section 3.1 Context expectations sit at a level of abstraction above the classes of network output, and are used in selecting affine transformations placed above the output of injection levels. When applied to injection level output, the affine transformations bias injection level feature representations towards conformance with the associated context expectation, illustrated in Figure[4] Injection level output is made more amenable to manipulation according to context through introduction of a contrastive loss Lo, introducing bias towards orthogonalization across context.
4

---

Published as a conference paper at ICLR 2023
Animate
A
ANIMTT
Non- Non-Anira Animale
Figure 4: Illustaaioon of the effect of affine transformation application over injection level features. Shown are images from two contexts, and the injection level feature representations for those con- texts. If the dimension of the feature map of the injection level is HxWxC (Height x Width x Chan- nel), this vector space is C-dimensional. The feature vectors are color coded according to context. After feature vector modulation through application of an affine transformation, the characteristics of the context associated with that affine transformation are more prominent. For example, with reference to Figure [ if it is known that the context is one involving animals, rather than food, then characteristics may biased towards fur interpretations, rather than kiwi fuzz or pastry surfaces.
Affine transformations applied over the features of the injection level for the purpose of amplify- ing or dampening certain characteristics. This process aligns mid-level representations towards con- formance with higher level context expectation. The affine transformations are made more effective through the disentanglement of characteristics at the injection level produced by the orthogonalizing bias.
During test time the CNN runs as single stream (without the connection across the streams of multiple samples which the contrastive loss introduces), and context expectation selects the affine transformation to apply over the feature vectors of the injection level. This expectation can come from any source in Section 4 we compare performance across network produced context expecta- tions and ground truth context expectations.
3.1 LOSS AND TRAINING
Training is broken into two stages, detailed in Table In the first, the base network is trained on its own, and features are biased towards orthogonality at the injection levels. In the second stage, the learning rate for the network parameters is reduced, and affine transformations are added to the injection levels according to the context categories input samples, initialized to identity matrices with added random noise, and given their optimizers and learning rate. In the first stage, gradients backpropogate through the base network only, bypassing the affine transformations: in the second stage gradients pass through both the base network and the affine transformations, In each stage we employ batches containing equal proportions of samples belonging to each context.
Training is broken into two stages for few reasons: 1) this division allows the possibility of using pretrained networks and training afline transformations in injection levels with no modification to the base network (X and 17,, 0), 2) allows fine-tuning of pretrained networks (A and/or Nx, 0), 3) we find that starting training of affine transformations after the feature representations have had chance to converge helps the affine transformations train intuitively, the affine transfor- mations have to adapt to less of moving target. We allow the network parameters to continue to train after affine transformations have been introduced at reduced learning rate as we find that this benefits performance.
To illustrate this process, consider a batch containing a horse and a desk image, horse belongs to animate while desk belongs to inanimate context. Both the images are fed in the same batch to the network during training. When training is in Stage 1, each image passes through the base network, sans affine transformations. The feature representations of the injection levels for horse and desk are then connected to each other via Lo. and biased towards orthogonality with respect to each other. When training progresses to Stage 2, the Lo contrastive connection is removed,
5

---

Published as a conference paper at ICLR 2023
and affine transformations inserted according to context expectation. Both network and affine transformation parameters are then updated according to gradients that pass through both affine transformations and network parameters.
We find that the angles between mid-level features associated with higher level contexts can be increased much greater degree than they would be without this orthogonalizing loss, measured by cosine similarity, without appreciable negative impacts on performance. See Figure 6 in the Appendix for an illustration of the extent which this cosine loss reduced when introducing this orthogonalizing loss.
Stage
Stage 2
Loss
=x CE((Y))
=
Optimizers
Ox(1n, .1)
ON L2) 12) OA., (B L2)
Table Definition of losses (fL2, L2}, and their use in optimizers across Stages and 2 of training. Y, and F correspond to sample labels, predictions, and injection level features, respectively. C is the context set. Lo (F, Y) is the orthogonalization bias loss described in Equation 1 (intermediate scaling term, and CF P) is cross entropy loss. Optimizer ON optimizes loss Lj according to learning rate Tly for training stage j. Optimizer O is an optimizer applied over only the parameters of the affine transformation associated with context C;, for network learning rates Uy, Nly affine transformation learning rate Te and losses Lr, L2.
loss scaling)
Lo(F,Y)
Σ
max(0, f1 f2
(1)
SSExl
||
Where
Spy = - -
(2)
Where
Fa. = (flc =c}
(3)
Where |Spyy is a method vector f was produced, I(f) is the injection level from which f was taken, and (AA) the uniform probability distribution over elements of A.
Yc(f) is the context of the sample from which feature
With Lo we wish to separate the angles of features associated with different contexts, in order to better enable manipulation through affine transformations. This can be seen as exacerbation of CNN's natural tendency towards decoupled representations where feature type has tendency to group according feature vector angle through structuring of characteristics' feature vector angles according to higher level context.
A.B We do this through cosine loss ( [B] applied to the features of the injection level. As wish control the network representation terms of context expectation, apply this loss across context. Figure in the Appendix illustrates the behavior produced by the orthogonalizing bias.
4 EXPERIMENTS
Here experiments demonstrating the utility of our feedback method. We perform eval- uations over CIFAR100 (Krizhessky et al. (009). ImageNet (Deng et al.] (209).. and the Caltech UCSD Birds Data (Birds) (Wah et al. 2011). all with multiple context splits these datasets

---

Published as a conference paper at ICLR 2023
CIFAR
ImageNet
Birds
3
Full
2
3
VGG
87_5
75.5
87.1
88.7
72.64
88.51
86.5
88.2
70.37
71.02
71.80
VGG (PF)
89.1
76.8
88.9
89
74.32
90
88.17
89.1
71.25
76.6
74.7
VIT
81.77
70.8
75.32
81.1
71.1
73.66
76.03
79.92
70.25
69.5
71
ViT (PF)
83.17
76.97
78.0
81.9
74.29
74.0
79.75
80.24
70.3
69.1
71.2
CNN
71.6
57.1
70.34
71.3
52
57.8
70.23
64.8
X
CNN (PF)
74.1
59.5
71.91
74
59.8
58.06
70.6
66
Table 2: Object classification accuracies (shown in percentages) across model and dataset splits. Comparison between base model and feedback based on Predicted Feedback (PF) context, where the context fed down the network is first predicted by the same network (accuracies for context prediction are given in Table i in the Appendix). The simple CNN was not applied over Birds data as its expected input resolution is small.
CIFAR
ImageNet 2
Birds
2
3
Full
VGG
87.5
75.5
87.1
88.7
72.64
88.51
86.5
88.2
70.37
71.02
71.80
VGG (GTM)
92.8
81.2
89.8
93.4
74.2
89.6
89.4
90.44
75.32
77.26
74.82
VGG (GTF)
93.4
81.4
88.8
93.8
74
89.8
89.07
91.36
78.77
79.23
76.17
ViT
81.77
70.8
75.32
81.1
71.1
73.66
76.03
79.92
70.25
69.5
71
VIT (GTM)
89.8
78.1
79.3
87.8
73
76.5
79.81
83.91
ViT (GTF)
90.22
78.4
79.2
89.9
73.1
76.6
77.66
80
CNN
71.6
57.1
70.34
74
52
57.8
70.23
64.8
CNN (GTM)
81.3
64.4
71.37
79.5
55.7
61.95
71.45
72.3
X
X
CNN (GTF)
83.2
64.4
71.9
80.7
62.1
62.85
73
71.2
x
x
Table Object classification accuracies across model and dataset splits. Comparison between base model, Masking based on Ground Truth context (GTM), and Feedback based on Ground Truth context (GTF). The simple CNN was not applied over Birds data its expected input resolution is small.
described in Section of the Appendix and derived based on the CIFAR-100 superclasses, and the attribute labels provided in the CUB dataset. We base splits on this information in order to eval- uate over standard divisions in the data. We evaluate our method using both ground truth context expectations (GT Feedback), structure as the base network (Pred Feedback). All experiments are conducted using 6-layer CNN base architecture, ct al. 2022), with variants including added affine transformations for feedback runs. The hyperpa- rameters to the modifications made to each of the base architectures for feedback incorporation described briefly in Section in the Appendix, and described detail in Section Din the Appendix. We show confusion matrices for 10-class context split over CIFAR 100 for both ground truth text expectations and network predicted context expectations, in Figures 5a and (5b] respectively. In Sections 4.1 and 4.2 evaluation involves comparisons base architectures tuned to maximize base architecture performance, and Section 4.3 presents an evaluation where both base architecture and feedback implementation are tuned maximize the benefit of feedback, giving insight into extent of potential benefit to feedback.
well as context expectations derived from a network of the same
V6 network (Simonyan & Zisserman, 2014), and a Transformer model (Tu
4.1 ORACLE EVALUATION
The ground truth context model (GT Feedback) assumes access to the ground truth context belong- ing each input sample during training and test time, using contextual knowledge to index into the affine transformations for application over mid-level features as described in Section 3 Here we evaluate the extent to which our complete framework outperforms 1) base architecture mirroring that our framework, and 2) the same base architecture where a hard masking operation using the ground truth contextual knowledge is applied over its class-level outputs. The GT Masking baseline corresponds to the same base architecture where a hard mask of c E {0, 1] is applied over the output of the network, where l corresponds to the number of classes and cg takes value of 1 for

---

Published as a conference paper at ICLR 2023
blcycle
bicccle
bus
motorcycle
motarcycle
pickup_tuc truck
tickup_truck
trinn
mower
swin mower
rocket
rocket
streetcar
streetccr
tank
tank
tractor
tractor
bloycle
bus
motorcyee
lown mowerr
rocket
steuttar
tank
tractor
bloyle
bus
motorcycle
trin
leon moooee
roclet
streetorr
tank
(a) With orthogonalization bias and affine transforma-(b) With neither orthogonalization bias nor affine trans- tions.
formations.
Figure 5: Confusion matrices for a our simple architecture (see Appendix with and without feed- back (orthogonalization bias and with application of affine transformations) subfigure a and respectively over the vehicles vs vehicles split of CIFARIOO 100. The first 5 rows correspond to the first context, and the next rows correspond the second context. Note that cross context confusion (quadrants and of the confusion matrices) is significantly reduced with feedback.
within-contex classes and for out-of-context classes. Results for ground truth context evaluation are shown in Figure
4.2 PREDICTED CONTEXT EVALUATION
Table 2 shows performance of base architectures with and without feedback, when context is pre- dicted rather than provided. Context prediction is performed through the addition of a second logits classification head to the base networks. This head is trained in conjunction with the object classifi- cation head using ground truth context labels. Performance for context prediction is shown in Table 6 in the Appendix.
4.3 MAXIMIZING FEEDBACK GAIN
We here present an evaluation where we tune for maximizing margins of feedback performance over base model performance, tuning both base model and feedback parameters. This evaluation provides insight into the degree to which feedback is capable of improving performance. We tune over the following hyperparameters to maximize the margins between feedback models relying on ground truth and the base models without feedback: first stage learning rate, weight decay, second stage learning rate, and affine learning rate. See the Appendix for precise values.
4.4 DISCUSSION
We evaluate the utility feedback under two scenarios: where context is known (Table 3). 2) where context is not known (Table 2). This provides insight both on the performance of the feed- back mechanism realistic cases with imperfect information derived from the same network. We observe consistently positive results in each case. We also compare feedback to a strong alternative of masking out (ex- cluding) predictions not associated with the ground truth context. Consistent with Figure superior performance of feedback over masking demonstrates that it improves modeling beyond simply re- moving context inconsistent predictions. Results architectures were tuned to maximize accuracy, and comparisons where base and feedback models tuned maximize the benefit of feedback. Margins much higher when tuning maximize
ideal cases (providing an upper bound on the utility of feedback), as well as
presented both in comparisons where base
8

---

Published as a conference paper at ICLR 2023
CIFAR
ImageNet
Birds
Full
VGG
87.1
70.4
83.8
85.9
70.5
85.3
85.0
85.4
70.0
72.8
68.4
VGG (PF)
89.9
77.4
85.8
88.2
77.0
88,9
86.1
89.5
75.6
77.4
77.8
VIT
82.1
56.8
78.7
82.4
65.4
74.0
77.0
78.6
66.6
54.2
52.7
VIT (PF)
86.9
59.0
82.6
85.7
68.0
76.0
79.2
83.9
69.9
65.1
64.3
CNN
67.5
55.9
66.1
63.2
49.9
53.6
55.9
58.4
CNN (PF)
77.1
60.3
70.6
71.3
52.9
58.0
61.9
63.9
Table 4: Results where we Maximize Feedback Gain (see Subsection 4.3 Object classification
accuracies across model and dataset splits, where comparisons are between base models and feed- back based on Predicted Feedback (PF) context, where the context fed down the network is first predicted by the same network. The simple CNN was not applied over Birds data its expected input resolution is small.
CIFAR
ImageNet
Birds
1
Full
VGG
87.1
70.4
83.8
85.9
70.5
85.3
85.0
85.4
70.0
72.8
68.4
VGG (GTM)
91.7
75.4
88.2
90.0
72.2
86.6
87.3
88.6
76.9
77.3
74.6
VGG (GTF)
93.3
81.5
88.4
91.9
76.4
88.8
87.5
91.1
80.2
79.1
80.4
VIT
82.1
56.8
78.7
82.4
65.4
74.0
77.0
78.6
66.6
54.2
VIT (GTM)
87.2
64.0
82.5
86.3
68.1
77.5
80.0
82.2
70.0
61.8
61.8
VIT (GTF)
91.1
66,2
85.7
90.5
71.9
79.5
81.7
86.0
73.9
65.1
65.6
CNN
67.5
55.9
66.1
63.2
49.9
53.6
55.9
58.4
X
CNN (GTM)
79.3
64.8
73.5
74.5
51.4
59.6
62.9
66.6
X
x
CNN (GTF)
82.4
69.0
75.6
76.9
56.0
62.2
64.7
69.8
X
Table 5: Results where we Maximize Feedback Gain (see Subsection 4.3
Object classification
accuracies across model and dataset splits, where comparisons are between base models, Masking based on Ground Truth context (GTM) and Feedback based on Ground Truth context (GTF). The simple CNN was not applied over Birds data as its expected input resolution small.
the benefit of feedback, and give insight into the extent to which feedback is capable of providing benefits the best case.
CONCLUSION
We have presented argument for the utility of feedback in vision. Feedback is 1) prominent in biological vision, with the majority of neural connections in the cortex consisting of feedback connections, 2) allows better constraining of under-constrained processes of abstraction, 3) allows for the online adaptation of vision systems towards alignment with high level understanding of the world. We leverage the fact that CNNs have tendency towards decoupled representations, exacerbating the separation of mid-level features associated with different higher level contexts. This allows better direct manipulation of the level which feedback introduced, minimizing collateral effects on characteristics not being selected for. In contrast to post-hoc filtering of interpretations for consistency with context expectations, MVF allows for cross-context detections and produces higher accuracies. MVF involves a top-down bridging of the signal-symbol gap, making it applicable to range of applications. In the future this work will be extended to localization. e.g. object detection or semantic segmentation, well used in embodied contexts Fermüller & Maynord (2022) with an active agent.
Fentexls/Frmüiller
REFERENCES
Ruzena Bajosy, Active perception. Proceedings of the IEEE, 76(8):666-1005 1988.
Isaac Caswell, Chuanqi Shen, and Lisa Wang. Loopy neural nets: Imitating feedback loops in the human brain. Tech. Report, 2016.

---

Published as a conference paper at ICLR 2023
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Cornelia Fermüller and Yiannis Aloimonos. Vision and action. Imuge and Vision Computing, 13 (10):725-744, 1995.
Comella Fermüller and Michael Maynord. Learning for action-based scene understanding. In Ad- vanced Methods and Deep Learning in Computer Vision, pp. 373 403. Elsevier, 2022.
Charles D Gilbert and Wu Li. Top-down influences visual processing. Nature Reviews Neuro- science, 14(5):350-333 2013.
Charles D. Gilbert and Mariano Sigman. Brain states: Top-down influences in sensory process- ing. Neuron, 5475)6677-666 2007. ISSN 0896-6273. doi: butp:://bb.org/100006...eemm 2007.05.019. URL https://wwat wtt.://eeeeeeeeeee.c//////eeeeee
S0896627307033765
Anne Harrington and Arturo Deza. Finding biological plausibility for adversarially robust features via metameric tasks. SVRHM 2021 Workshop@ NeurIPS, 2021.
Hu and D. Ramanan. Bottom-up and top-down reasoning with hierarchical rectified gaussians. In 2016 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 5600-5609, Los Alamitos, CA, USA, jun 2016. IEEE Computer Society. doi: 10O.///......../.6/0 URL
https://dee http:////...accaooeoeoeeeeeee///
2016. 604
Akshay V Jagadeesh and Justin Gardner. Texture-like represennation of objects in human visual cortex. Proceedings of the National Academy Sciences, 119(1))e211530211 2022.
Talia Konkle and George A Alvarez. Instance-level contrastive learning yields human brain-like representation without category-supervision. BioRxiv, 2020.
Gabriel Kreiman and Thomas Serre. Beyond the feedforward sweep: feedback computations in the visual cortex. Annals of the New York Academy of Sciences, 1464(1):222-241, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive cluster- ing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8547-8555, 2021.
Qianli Liao and Tomaso Poggio. Bridging the gaps between residual learning. recurrent neural networks and visual cortex. CoRR, abs/1604.00364, 2016. URL httt:://aaaee /ttp///aavvv///a//////eee/aaaev/ org /bs/
1604. 03640
Drew Linsley, Alekh Karkada Ashok, Lakshmi Narasimhan Govindarajan, Rex Liu, and Thomas Serre. Stable and expressive recurrent vision models. Advances in Neural Information Processing Systems, 33:10456-11444 2020a.
Drew Linsley, Junkyung Kim, Alekh Ashok, and Thomas Serre. Recurrent neural circuits for tour detection. arXiv preprint arXiv:2010.1531 2020b.
Weiyang Liu, Zhen Liu, Zhiding Yu, Bo Dai, Rongmei Lin, Yisen Wang, James M. Rehg, and Le Song. Decoupled networks. CoRR, abs/1804.08071,, 2018. URL httt::://axxiivorrgg
abs 1 1804. 08071
Bria Long, Chen-Ping Yu, and Talia Konkle. Mid-level visual features underlie the high-level cate- gorical organization of the ventral stream. Proceedings of the National Academy of Sciences, 115 (38):99015-E9024, 2018
William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video pre- diction and unsupervised learning. arXiv preprint arXiv: 1605.08144 2016.
10

---

Published as a conference paper at ICLR 2023
Nikola T Markov, Maria M Eresey-Ravasz AR Ribeiro Gomes, Camille Lamy, Loie Magrou, Julien Vezoli, Pierre Misery, Arnaud Falchier, Rene Quilodran, Marie-Alice Gariel, et al. weighted and directed interareal connectivity matrix for macaque cerebral cortex. Cerebral cortex, 24(1): 17-36, 2014.
NV Kartheek Medathati, Heiko Neumann, Guillaume S Masson, and Pierre Korproobst. Bio- inspired computer vision: Towards a synergistic approach of artificial and biological vision. com- puter vision and image understanding, 150:1-30. 2016.
Morimichi Nishigaki, Cornelaa Fermüller, and Daniel Dementhon. The image torque operator: A new tool for mid-level vision. In 2012 IEEF Conference Computer Vision and Pattern Recog- nition, pp. 502-509. IEFF, 2012.
Sofia Paneri and Georgia G. Gregoriou. Top-down control of visual attention by the prefrontal cortex. functional specialization and long-range interactions. Frontiers Neuroscience, 2017. ISSN 1662-453X. doi: 20..8////nnss22111.00000.
URL https:/ h/tp::///wwwt. www. www.:Eroneee frontiersin.
org/article/ / fn 2017. 00545
Pedro Pinheiro and Ronan Collobert. Recurrent convolutional neural networks for scene labeling. In Eric P. Xing and Tony Jebara (eds.), Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp. 82-90, Be- jing, China, 22-24 Jun 2014. PMLR. URL https: ptttp:://proceedinggs mwt.sss..rr.g. wtt.eedd.nggmm....eess//////2 pinheiro14 html
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint MeXXi::4409.380 2014.
Hanlin Tang, Martin Schrimpf, William Lotter, Charlotte Moerman, Ana Paredes, Josue Or- tega Caro, Walter Hardesty, David Cox, and Gabriel Kreiman. Recurrent computations for visual pattern completion. Proceedings of the National Academy of Sciences, 15(35):8835 8840, 2018.
Ching Teo, Cornelia Fermuller, and Yiannis Aloimonos. Fast 2d border ownership assignment. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5117- 5125, 2015a.
Ching L Teo, Cornelia Fermuller, and Yiannis Aloimonos. Detection and segmentation of 2d curved reflection symmetric structures. In Proceedings of the IEEE International Conference on Com- puter Vision, pp. 1644-1652, 2015b.
Ching L. Teo, Comelia Fermüller, and Yiannis Aloimonos. A Gestaltist approach to contour-based object recognition: Combining bottom-up and top-down cues. The International Journal of Robotics Research, 34(4-5):627-652. 2015c.
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. Maxvit: Multi-axis vision transformer. arXiv preprint arX1v:2204.01697, 2022.
Ruben S van Bergen and Nikolaus Kriegeskorte. Going in circles is the way forward: the role of recurrence in visual inference. Current Opinion in Neurobiology, 65:176-193, 2020.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Yong Xu, Yuhui Quan, Zhuming Zhang, Hui Ji, Comelia Fermüller, Morimichi Nishigaki, and Daniel Dementhon. Contour-based recognition. In 2012 IEEE Conference on Computer Vision and Pattern Recognition. pp. 3402-3409. IEEE, 2012.
Jian Yao, Sanja Fidler, and Raquel Urtasun. Describing the scene as a whole: Joint object detection, scene classification and semantic segmentation. In 2012 IEEE conference computer vision and pattern recognition, pp. 702-709. IEEF, 2012.
A. R. Zamir, T.-L. Wu, L. Sun, W. Shen, B. E. Shi, J. Malik, and S. Savarese. Feedback Networks. ArXiv e-prints, December 2016.
11

---

Published as a conference paper at ICLR 2023
CIFAR
ImageNet
Birds
2
3
4
Full
3
1
2
3
VGG
91.65
95.34
94.62
95.11
96.43
98
97.9
97.8
81.19
90.3
75.3
ViT 90.32
96.17
93.46
94.17
96.2
98.5
94.3
97
80.4
87.19
80.29
CNN
82.9
85.5
88.24
85.5
87.25
89.33
90
92.46
x
X
X
Table 6: Context classification accuracies (shown as percentages, and used in Section 4.2) across model and dataset splits. The simple CNN was not applied over Birds data as its expected imput resolution is small.
A APPENDIX
ABSTRACT
We here present Appendix. Section describes datasets; Section D presents the base CNN architecture used in experiments; Section | details tuned parameters; and, Section K presents data splits.
B DATASETS
B. CIFARIO0
We adopt CIFAR100 for the 1) high-levels of visual ambiguity due to low resolution, 2) existence of several distinct "superclasses" consisting of a roughly equal number of classes, and 3) the cross- context confusion across classes highly similar in appearance (e.g., sharks and dolphins). We adopt the official training and test split for the CIFAR 100 dataset. Each class contains exactly 500 training images and 100 testing images, with each superclass consisting of 5000 training images and 1000 testing images. We use the CIFAR100 superclasses in constructing context splits. Split I: Vehicles 1 Vehicles 2, Split 2: Household Devices Furniture, Split 3: Aquatic Mammals Fish. For full class breakdown see Appendix Section K
B.2 IMAGENET
We adopt ImageNet for several of the aforementioned reasons above, as well its generality in that it spans 1000 unique classes. Each class contains variable number of images we designate 80% of the images in each class for training, but only 2% for testing due the computational cost incurred by the high number of images and the need for frequent testing.
We employ the following context split over ImageNet, designed to be similar CIFAR100 splits, the full class breakdown of which is given in Appendix: Split I: Household Devices vs. Furniture, Split Aquatic Mammals vs. Fish, Split Vehicles vs. Vehicles 2.
C BASE CNN
Figure 8 illustrates the base CNN model (apart from VGG and ViT), for which performance is reported in Tables in the main paper.
D CONTEXT PREDICTION
See Table 6 for accuracy on context prediction used in Section 4.2 and Table see Table for accuracy on context prediction used in Section 4.2 and Table 2
E IMAGENET-C EVALUATION
T provides accuracy testing on ImageNet-C, with models trained for Maximizing beek Gain over standard ImageNet. Accuracy trends consistent with trends presented in Section 4.3
12

---

Published as a conference paper at ICLR 2023
CIFAR
ImageNet
Birds
1
2
3
4
Full
2
3
2
3
VGG
93.7
91.9
89.6
90.4
96.1
97.8
95.9
97.6
79.0
89.4
80.3
VIT
90.32
96.17
93.46
94.17
96.2
98.5
94.3
97
80.4
87.19
80.29
CNN
82.3
86.1
88.8
81.0
90.2
85.6
87.6
90.6
X
X
X
Table Context classification results of the runs Maximizing Feedback Gain (see Subsection across model and dataset splits. The simple CNN was not applied over Birds data its expected input resolution small.
4.3
ImageNet
3
CNN
39.5
41.8
42.8
CNN PF
42.3
47.6
48.0
CNN GTM
45.5
48.7
52.4
CNN GTF
47.7
50.1
53.9
VIT
58.3
61.7
64.1
ViT PF
61.3
64.8
69.5
VIT GTM
62.1
64.6
66.5
VIT GTF
64.4
66.4
68.5
VGG
69.7
69.3
69.8
VGG PF
74.5
71.8
75.3
VGG GTM
70.9
72.3
73.2
VGG GTF
73.3
71.9
75.2
Table 8: Object classification accuracies across ImageNet-C, where the training set is unmodified and various corruptions are applied over the test set. Comparison between base model, Masking based Ground Truth context (GTM), feedback based Predicted Feedback (PF) context, and Feedback based on Ground Truth context (GTF).
ImageNet-C consists of 75 common corruptions applied over ImageNet images with the intent of degrading classifier performance. We observe that drops in accuracy with respect to the original ImageNet dataset range between values of 14% and 18%. However, performance margins between feedback and base models are overall maintained when testing over ImageNet-C.
DECOUPLED REPRESENTATIONS
CNNs have natural tendency towards decoupled representations. These are representations where characteristies have a tendency to represented such way that feature vector angle corresponds to characteristic type, while feature vector magnitude corresponds to characteristic variation or de- gree (Liu et al. 2018
G ORTHOGONALIZING Loss
Figure 7 illustrates feature vector projections of the injection level under different degrees of orthog- onalizing loss.
13

---

Published as a conference paper at ICLR 2023
X
LOSS
Training lterations
Figure 6: Shown above is convergence plot of orthogonality loss Lo over several (intermediate loss scaling) values (0, 0.075, 1,5).
(a) PCA Projection,
(h) PCA Projection,
(c) PCA Projection, 50
Figure 7: Illustrations of the effect of different levels of X, the scaling term applied to orthogonal- ization bias Lo. applied to an injection level of the second to last conv op of the architecture given in the Appendix. Feature vectors of size 64 for the (animate) / (inanimate) split (see Appendix) of CIFAR100 are projected into dimensions through Principal Component Analysis (PCA). Blue points correspond features produced from animate images, and red points correspond to features produced by inanimate classes. Subfigure illustrates context distribution when no orthogonalizing bias used 0) there some degree of context separation, but it not significant. moderate degree of orthogonalizing bias (A = 2) in subfigure 2 separates animate from inanimate features in the injection level. significant degree of orthogonalizing bias (A 50) results poor separation and degraded performance. See Appendix for more details, including UMAP projections.
Convau
Cam2D
Convzu
Dense
ReLU
ReLUU
ReLU
ReLU
Conv2D
Com/D0
Conv2D
Dropout
ficatinication
ReLU
ReLu
f
Dropout
Dropout
Drapout
ReLU
Figure 8: Illustration of the base architecture (SimpleCNN) used in these experiments. The injection level (circled in red) used in these experiments after the ReLU the second to last Conv2D operator.
14

---

Published as a conference paper at ICLR 2023
H MODELS
Here we describe the architectures in which we incorporate feedback. Each model consumes GPU during train and test time. For all feedback experiments we choose À (intermediate loss scaling) of 1.0 (otherwise set to 0.0).
Shallow CNN: This model comprises 6-layer CNN architecture, shown in Appendix, consisting of 3 by 3 shaped kernels, max-pooling applied over every other layer, and dropout (p = 0.375, p 0.1) applied over the penultimate fully connected layer and after each convolution operation, respectively. The affine transformation is applied after the second to last convolution operation, though we observe high performance inserting the affine transformations anywhere throughout the second half of the architecture. We train the first stage for roughly 5 million iterations for all splits. learning rate of 0.001 is chosen for the training of the base network during the first stage, and learning rate of 5 * 10-5 is chosen for the learning rate of the base network during the second stage, whereas the affine transformation learning rate set to * 10-3
For the Maximizing Feedback Gain hyperparameters, we adopt learning rate of 2e - 4, a weight decay of 0.0, a second stage learning rate of 1 6, and an affine learning rate of 0.005.
VGG: Here we adopt VGG-16 network with pre-trained weights over ImageNet. The VGG net- work consists of 16 layers consisting of convolution and max-pooling operations. The affine trans- formation is applied after the eleventh convolution operation, though we observe high performance inserting the affine transformations anywhere throughout the last six layers. We train the first stage for roughly 1.5 million iterations for all splits, until smooth convergence. A learning rate of 5 10 is chosen for the training of the base network during the first stage, and a learning rate of 2.5 * 10 is chosen for the learning rate of the base network during the second stage, whereas the affine trans- formation learning rate is set to 2.5 * 10-3
For the Maximizing Feedback Gain hyperparameters, we adopt learning rate of 5e 5, weight decay of 0.00075, a second stage learning rate of 5e - 6, and an affine learning rate of 0.0005.
Here adopt variant of the Visual Transformer models (Tu et al. 2022). general-purpose vision transformer that outperforms many related visual transformer architectures while being easy to train. The affine transformation is applied immediately after the third last attention block. We train the first stage for roughly 2.0 million iterations for all splits. until smooth convergence. A learning rate of * 10-3 3 is chosen for the training of the base network during the first stage, and learning rate of 1.0 10-6 is chosen for the learning rate of the base network during the second stage, whereas the affine transformation learning rate is set * 10
Visual Transformer:
For the Maximizing Feedback Gain hyperparameters, we adopt learning rate of 2e - 4. a weight decay of 0.0, a second stage learning rate of le and an affine learning rate of 0.0001.
PARAMETERS
We here list parameters' tuned values not introduced in the main paper:
1. Image size: 32 x 32 for CIFAR100, 224 x 224 for ImageNet.
2. Model input image size: 32 32 for 6-layer CNN, 224 x 224 for VGG16. Images resized using bilinear interpolation.
3. Size of feature set selected for orthogonalization: 25.
4. Batch size: 256 (CIFARI00 splits), 64 (ImageNet splits).
5. Data augmentations: Random rotations (15 degrees), random resized crops, Random hori- zontal flips.
6. Feedback Base Model: ADAM's optimizer, weight decay 7.5 x 10-4 for both stages and both models.
7. Affine Transformation Optimizer: ADAM's optimizer, affine transformation learning rate of 0.001 for second stage training of both models.
8. Context Model: ResNet18 model with pretrained weights over ImageNet and learning rate of 0.001 using SGD optimizer.
15

---

Published as a conference paper at ICLR 2023
CIFAR
ImageNet
Birds
Full
VGG
87.1
70.4
83.8
85.9
70.5
85.3
85.0
85.4
70.0
72.8
68.4
VGG (*)
87.4
70.8
84.6
87.3
71.7
86.8
85,8
86.0
71.2
73.9
70.0
VGG (**)
87.7
70.4
86.0
87.5
72.0
88.0
85.4
86.7
71.9
74.2
71.3
VGG (Pred Feedback)
89.9
77.4
85.8
88.2
77.0
75.6
77.4
VGG (GT Feedback)
93.3
81.5
88.4
91.9
76.4
88.8
87.5
91.1
80.2
79.1
80.4
ViT
82.1
56.8
78.7
82.4
65,4
74.0
77.0
78.6
66.6
54.2
52.7
VIT (*)
84.7
56.9
79.8
83.1
65.8
74.6
75.3
79.5
67.0
56.2
56.7
ViT (ot)
85.2
57.1
80.0
83.0
67.0
75.9
75.7
80.0
67.8
ViT (Pred Feedback)
86.9
59.0
82.6
85.7
68.0
76.0
79.2
83.9
69.9
65.1
64.3
VIT (GT Feedback)
91.1
66.2
85.7
90.5
71.9
79.5
81.7
86.0
73,9
65.1
65.6
CNN
67.5
55,9
66.1
63.2
49.9
53.6
55.9
58.4
CNN (*)
71.3
55.9
67.4
58.5
CNN (**)
72.9
57.0
67.8
67.1
50.5
56.2
55.9
59.3
X
CNN (Pred Feedback)
77.1
60.3
70.6
71.3
52,9
58.0
61.9
63.9
CNN (GT Feedback)
82,4
69.0
75.6
76.9
56.0
62.2
64.7
69.8
Table 9: Object classification accuracies across models (Maximizing Feedback Gain, Section 4.3) and dataset splits. We additionally report numbers when the context training signal is included in training (**) vs. when the context signal is excluded from training (*), when the feedback mecha- nism not incorporated (neither the affine selection nor the orthogonalization loss). This isolates the impact of the inclusion of the context label during training while removing the impact of the feedback mechanism. The difference in performance between (*) and (**) runs is attributable to the inclusion of the context signal during training. For both runs we use affine learning rate of 0 and a single affine transformation as opposed to N affine transformations (where N is the number of context divisions;
single affine to avoid introducing a context signal, even when affines are not trained). context scaling loss value of 0.0 is used for (*) and a value of 1.0 for (**). The runs of (#*), where the context signal is included but feedback is excluded, underperform w.r.t. where feedback is included.
J CONTEXT LABEL, AFFINE LEARNING RATE, ORTHOGONALIZIIN Loss ABLATION
In Table evaluate the effect performance due simply the introduction of the affine trans- formation (and the random noise introduced by its introduction). but not due to the context training labels. We report numbers from experiments where the affine operations but: affine transformations are not trained. the context prediction head is not trained, and orthog- onalizing loss is not employed. These runs are compared against identical runs where the affine transformation is not included. We observe that runs with affine transformations outperform the results of the base models (where no affine transformations are included), for two main possible reasons: 1) The drop in learning rate during the second stage of training allows accuracy to tinue converging after possible plateauing, and 2) The introduction randomly initialized affine during the second stage introduces stochasticity potentially useful during training. This increase in performance small comparison to the increase due to incorporation of feedback.
included in the network
K
DATA SPLITS
We derive context splits based the superclass structure provided with CIFAR-100 (over both CIFAR- 100 and ImageNet), and the attribute ontology provided with the CUB dataset. We base splits this information in order evaluate over standard divisions in the data.
K.1 CUB-200-2011
We adopt the Caltech-UCSD-EE UCSD-Birds dataset for several of the aformentioned reasons above, in partic- ular for the high cross-context confusion across different species of birds highly similar in appear- ance. It consists of 1 11,78 images with 200 classes corresponding to bird species. Like the Imagenet dataset, we designate 80% of the dataset for training and 20% for testing.
16

---

Published as a conference paper at ICLR 2023
We employ the following splits over the CUB dataset, grouping images into contexts based on the listed attributes provided with the CUB dataset:
Migration behavior (1, 2, 3)
2. Trophic level (Carnivore, Herbivore, Omnivore)
3. Primary lifestyle (Aerial, Aquatic, Generalist, Insessorial, Terrestrial)
K.2 SPLITCIFAR
CIFAR100 Dataset Sub-Splits
Split
Group
Classes
Vehicles Vehicles 2
Bicycle, Bus, Motorcycle, Pickup truck Lawn mower, Rocket, Streetcar, Tank, Tractor
2
Household Devices
Clock, Keyboard, Lamp, Telephone, television Bed, Chair, Couch, Table, Wardrobe
Furniture
Aquatic mammals Fish
Beaver, Dolphin, Otter, Seal, Whale aquarium fish, flatfish, ray, shark, trout
Small animals Large animals
fox, porcupine, possum, raccoon, skunk bear, leopard, lion, tiger, wolf
Full CIFARI00 Split:
animate = beaver, dolphin, otter, seal, whale, aquarium fish, flatfish, ray, shark, trout, bear, leopard, lion, tiger, wolf, camel, cattle, chimpanzee, elephant, kangaroo, fox, poreupine, possum, raccoon, skunk, baby, boy, girl, man, woman, crocodile, dinosaur, lizard, snake, turtle, hamster, mouse, rabbit, shrew, squirrel, bec, bectle, butterfly, caterpillar, cockroach, crab, lobster, snail, spider, worm
inanimate orchid, poppy, rose, sunflower, tulip, bottle, bowl, can, cup, plate, apple, mush- room, orange, pear, sweet.pepper clock, keyboard, lamp, telephone, television, bed, chair, couch, table, wardrobe, bridge, castle, house, road, skyscraper, cloud, forest, mountain, plain, sea, maple tree, oak tree, palm tree, pine tree, willow tree, bicycle, bus, motorcycle, pickup truck, train, lawn mower, rocket, streetcar, tank, tractor
17

---

Published as a conference paper at ICLR 2023
K.3 SPLITIMAGENET
ImageNet Dataset Splits
Split
Group
Classes
analog clock, digital clock, wall clock, computer key dial telephone, table lamp, television, cellular telephone studio couch, dining table, wardrobe, folding chair
Household Devices
Furniture
Aquatic mammals Fish
Beaver, Dolphin, Otter, Seal, Whale barracouta, eel, coho, rock beauty, anemone fish, sturgeon, gar, puffer, lionfish
mountain bike,
bus, moped, tricycle, bullet train, passenger car, pickup lawn mower, tractor, streetcar, tank
3
Devices
Devices
18


Reviews:
[{"review_id": "uEF643I-XlX", "reviewer": "Reviewer_zASa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a mid level vision feedback module as an Add-On for CNN's. Authors argue that such mid-level feedback properties improve CNN performance (though perhaps for the wrong reasons), and also argue that this idea is worth pursuing given neuroscience/perceptual psychology motivated ideas.", "summary_of_the_review": "++ Additional Controls and tentative confounding variables:\n\nTables 2 and 3 at first glance seem quite exciting and one could reasonably say that feedback is playing a critical role here. However, how can we dissociate between the effects of feedback and the effects of SSL or noise injection in training when comparing to the \"vanilla\" VGG, CNN and Transforer models? If a feedback loop would be the only other mechanism added in training then these tables are good indicators for the feedback effects in the paper, but if not, then knowing the exact contribution to the differential offset is not clear. Consider the question: what would have happened if a regular feedforward (withour recurrence) CNN would have been trained with noise injections, and PCA? Would the authors still get those positive bumps in score?\n\n++ Several Missing References that hurt the paper in relation to where it lies on innovation:\n\nMid-level vision:\n* Long, Yu & Konkle : \"Mid-level visual features underlie the high-level categorical organization of the ventral stream\" (texforms). PNAS 2019.\n* Jagadeesh & Gardner. \"Texture-like representation of objects in human visual cortex\" . PNAS 2022.\n* Harrington & Deza. \"Finding Biological Plausibility for Adversarially Robust Features via Metameric Tasks\". ICLR 2022.\n\nRecurrence:\n* Tang et al.: \"Recurrent computations for visual pattern completion\". PNAS 2018.\n* Lotter, Kreiman & Cox: \"Pred-Net\". ICLR 2017.\n\nContrastive Learning:\n* Konkle & Alvarez. \"Instance-level contrastive learning yields human brain-like representation without category-supervision\". ArXiv 2020 & Nature Communications 2022.\n* Li et al. \"Contrastive Clustering\". AAAI 2021.\n\n++ Additional experiments:\n\nIn experiments suggested by Long et al, Jagadeesh & Gardner, & Harrington & Deza, authors played around with machine vision classifiers showing them \"texforms\", these stimuli are stimuli that were synthesized from noise that match the local texture statistics of an object, in addition to preserving it's coarse shape. I wonder, if the argument here is to be made that mid-level feedback will improve general consistency and image categorization removing higher-order contextual biases, then the MVF-based networks should show similar confusion matrices for such texform stimuli and also the original stimuli. \n\nWhile it may be too difficult for authors to add these experiments in the rebuttal, it would be nice to see these or some variation such as common corruption robustness (ImageNet-C) or adversarial robustness (either quantiatively or qualitatively : see for example Berrios & Deza, ArXiv 2022 on testing 'brain-alignment' of transformer based models). \n\n+ Final Thoughts:\n\nOverall, I don't think having a small one or 2 point percentage increase in classificatiion accuracy (albeit a very coarse control, unless I missed something) is enough to convince me that \"feedback is better/necessary for computer vision\". I am however on the author's side, and I believe their premise is true, but I don't think these experiments drive the point home completly.\n\n", "strength_and_weaknesses": "See below for Main Paper Summary. TLDR: this paper studies mid-level vision as an improvement gateway for modern CNNs, however I do not think the baselines or experiments are well controlled enough to validate the statement. In addition it is not obvious what the contributions to performance are that stem from noise injections, recurrence and SSL-like learning when isolated, and not stacked together. In addition authors only focus on performance and not other (perhaps more interesting) tasks that extend object recognition such as common corruption robustness or adversarial robustness.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "nqaTffxbtos", "reviewer": "Reviewer_ym4H", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This is an interesting study that proposes novel ideas on how to incorporate contextual cues in the form of category-specific feedback signals into visual recognition algorithms. The proposed algorithm (mid-vision feedback, MVF) outperforms several baseline models in several relevant benchmark object recognition tasks. More generally, this paper introduces plausible mechanisms by which feedback signals (here in the form of expectations) could be incorporated into the traditional architectures for processing visual information.\n\n", "summary_of_the_review": "This is an exciting effort to bring ideas about how feedback signals can be incorporated to help incorporate contextual cues into visual recognition. Although the clarity of the presentation can certainly be improved quite a lot, the results and ideas are novel and help push the field forward.", "strength_and_weaknesses": "Strengths\n\nThe proposed architecture is quite simple and therefore elegant. \n\nThe proposed ideas about how to incorporate feedback are rather generic and can be directly applied not just for any one specific dataset or task, but rather across multiple different problems including cases of no contextual cues, cases where contextual cues are incongruent or unusual, cases where contextual cues are provided by task demands or by statistical correlations previously learned from images.\n\nWeaknesses\n\nSeveral of the points below are questions, rather than weaknesses.\n\nIt is true that the majority of connections are feedback rather than feedforward, but not because of the work of Kveraga et al. In humans, we do not know any details about anatomical connectivity. Probably the best work making this point is Markov et al Cerebral Cortex 2014. \n\nI find Figure 1 to be somewhat confusing. First, the authors state that they cropped the images to exclude context. But then they end up arguing that understanding the difference of context can help appreciate the low-level feature differences. Which one is it, is there context in these images or not? And does it help or not? \n\nFigure 2. What are the injection sites? What are the big blue and red “A” symbols, are those the affine transformations? Are L_O and L_0 the same thing? Where do the high-level contextual expectations come from? \n\nWhat are the x-axis and y-axis in Figure 5?\n\nWhile this is not the main point of this study, why is it that ViT generally performs below VGG in Tables 3 and 4?\nThe rationale for having two stages in training is not clearly explained. Why two stages as opposed to a single stage of training?\n\nRelated to the question above regarding Figure 2, what decides which representations should be orthogonalized? In the example of desk and horse, who in the model says that there should correspond to two different contexts and should be orthogonalized? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "0Mmm7PEjeS", "reviewer": "Reviewer_ZiEu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes that images will be better processed if the context is known. Figure 1 provides excellent motivation for this idea. The images are very similar in appearance, but correspond to very different classes (food vs. animals). Inspired by the well-known fact that there are as many or more feedback connections in the visual system as feedforward ones, the paper proposes the idea that this feedback from higher levels can provide context information, altering the features to be more aligned with that context. So the idea of the paper is to provide biasing feedback to the mid-level features of the network, emphasizing one context (stretching a vector representing that context, while shrinking another). In order to avoid too much interference between feature dimensions when this transformation is applied, the network first is trained with an orthogonality bias at the particular layer where the context transformation will be applied (termed the \"injection layer\").  Then it is trained to use a biasing context, which is actually a learned affine transformation of the features at that layer of the network, controlled top-down. Hence, if the context is known, this can be applied dynamically, depending on the context. This contextual \"hint\" improves classification performance of the network. They note that this context manipulation could come from a more symbolic level of processing, providing a link between convnets and neuro-symbolic processing, although that itself is not demonstrated. They also show that post-hoc filtering of the results (masking off animal outputs in a food context, for example) doesn't work as well as altering the feature dimensions at a hidden layer. They show that this manipulation is relatively generic, and can be applied to a vision transformer as well as a convnet, improving performance in both cases.\n\nI have read the authors' responses to my review, the other reviews, and skimmed the revised paper, and have bumped up my score by one point as a result.", "summary_of_the_review": "This paper suggests a novel method for providing contextual feedback to a network that is performing categorization, inspired by the biology that there are many feedback connections in cortex. So the motivation is good. The results are that performance is almost always improved by context feedback. However, the actual implementation of this is a bit unclear. Are there two matrices for the affine transformation, one for one context and one for the other? I don't recall seeing this in the paper anywhere. It also isn't clear whether there is a matched number of parameters between the networks with and without contextual feedback. For these reasons, my enthusiasm for the paper is somewhat reduced.", "strength_and_weaknesses": "Strengths:\n\n+ This is a relatively novel idea about how to use top-down connections to improve performance in particular contexts. I have not seen any other work like this. \n\n+ The context manipulation almost always improves classification performance. \n\n+ It seems the context can be discovered automatically by the same network that is performing the classification, but the system usually works better if the ground truth context is known, which could come from some other process providing contextual feedback. \n\n+ The approach is applied to a cross product of three different datasets and three different network architectures (except one cell is missing for a reasonable reason...), and almost always improves performance. \n\n+ The approach is shown to work better than post-hoc filtering of the non-contextual outputs, at least when the ground truth context is known, but not when the context is predicted automatically. \n\nWeaknesses, with concrete, actionable feedback\n\n- The writing could be clearer\n\n- It isn't clear whether the number of parameters is matched between the networks with feedback (which adds the parameters of the affine transformation) and without feedback. I assume this would add n^2 parameters, if there are n units in the layer.\n\n- The context splits seem particularly arbitrary (e.g., streetcar and rocket are in one context?). It would be good use (for example) Figure 1 as a running example, and have a context that has food and animals or something similar and show that those images don't get confused in context.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster