Asynchronous Perception Machine for Efficient  Test Time Training
Authors: Rajat Modi
Yogesh S Rawat
Venue: NeurIPS 2024 poster (2024)
Keywords: MORTAL COMPUTATION
GLOM
test time training
neural fields
implicit representation
distillation
Primary area: deep_learning_architectures
TL;DR: APM introduces asynchronous patch-processing for test-time-training instead of parallel perception. APM leverages GLOM's islands of agreement. We present the scientific evidence towards validating GLOM's insight:  if input percept is really a field.

Abstract: In this work, we propose Asynchronous Perception Machine (APM), a computationally-efficient architecture for test-time-training (TTT). APM can process patches of an image one at a time in any order asymmetrically and still encode semantic-awareness in the net. We demonstrate APM's ability to recognize out-of-distribution images without dataset-specific pre-training, augmentation or any-pretext task. APM offers competitive performance over existing TTT approaches. To perform TTT, APM just distills test sample's representation once. APM possesses a unique property: it can learn using just this single representation and starts predicting semantically-aware features.
APM demostrates potential applications beyond test-time-training: APM can scale up to a dataset of 2D images and yield semantic-clusterings in a single forward pass. APM also provides first empirical evidence towards validating GLOM's insight, i.e. input percept is a field. Therefore, APM helps us converge towards an implementation which can do both interpolation and perception on a shared-connectionist hardware. Our code is publicly available at  https://rajatmodi62.github.io/apm_project_page/

--------

**It now appears that some of the ideas in GLOM could be made to work.**

https://www.technologyreview.com/2021/04/16/1021871/geoffrey-hinton-glom-godfather-ai-neural-networks/

GLOM = Geoff's Latest Original Model.
```
   .-""""""-.
 .'          '.
/   O      O   \
|      O       |
 \  '------'  /
  '.        .'
    '-....-'
Silent men in deep-contemplation.
Silent men emerges only sometimes.
Silent men love all.
Silent men practice slow science.
```

Asynchronous Perception Machine for Efficient Test Time Training
Rajat Modi
Yogesh Singh Rawat
Centre for Research in Computer Vision
University of Central Florida
Orlando, FL. 32765
rajat.mooi@uue yow.eeeeeeddduooooooeoooeoooo ucf edu http://mnaammoodd.ggggioiooooppp
Abstract
In this work, propose Asynchronous Perception Machine (APM), computationally-effiiinn architecture for test-time-training (TTT). APM can pro- cess patches of an image one time in any order asymmetrically, and still encode semantic-awarenes out-of-distribution images without dataset-specifie pre-training augmentation any-pretext task. APM offers competitive performance over existing TTT ap- proaches. To perform TTT, APM just distills test sample's representation once. APM possesses a unique property: it can learn using just this single representation and starts predicting semantically-awart features.
in the net. We demonstrate APM's ability to recognize
APM demostrates potential applications beyond sest-time-training APM can scale up dataset of 2D images and yield semantic-clusterings in a single forward pass. APM also provides first empirical evidence towards validating GLOM's insight, i.c. input percept is field. Therefore, APM helps us converge towards an implementation which can do both interpolation and perception on a shared- connectionist hardware. Our code publicly available at this link
Introduction
In these past centuries, computing machines have become a lot faster [2]. This made it possible to train higher-pe saaher-parameterizeee neural nets and led to interesting emergent abilities [80]. As was predicted by Turing himself, and as were his suspicions of Lady Lovelace's 's arguments against learning machines 990: neural-nets can now finally learn without human-feedhack (6) paint pictures (76 and even compose a sonnet [92]. Even with such impressive-progress
a key question still remains: how can these nets recognize images whose distribution is far different from the ones which were used during training? For e.g., consider a self-driving car trying to stop when it encounters pedestrian crossing road. Such practical scenarios require iinstattaecous-deciiions for ensuring human-safety in autonomous-systems (64]
cost-time-training (TTT) (85) is one of the promising techniques for handling such distribution shifts: a neural net adapts to test sample 'on the fly'. Since the label of the sample is not known, the net
performs some auxiliary pre-text task like data augmentation [15]. rotation [15 86) or prompt tuning
84 on it. After several such iterations, the net recognizes the test sample. The key idea is that the net
allowed to dynamically adjust its decision boundary even after it has been trained, thereby bringing
much closer to how humans keep learning 'continuously" throughout their lifespans [86].
Coorreppnndecce trajatmodi 620gmmai...1......mmmm.m com
AA tlick of the eye, the fall of her hair, the passing of a second, or capturing the world at 120fps: instantaneous can be subjective experience. Seconds can sometimes pass slowly for other reasons also.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

"A Photo of Mona Lisa!
Zero Shot
ft
RGB,
Textual Encoder
Bennss
Encoder
Distiliatoon
Averaging
MLP
MLP
Sequential Querying of MLP
TiE Unfolded State
CNN filter
Flatten
T Folded > Unfolded
Pay
MLP
Gather Grid
T
T:
Flatten
Tn-jUntoided
RGB
Lay
(1) Asynchronous Perception Machine
(ii) Folded State
(iii) Unfolded State
Figure 1: (i) Asynchronous Perception Machine (APM): An image / passes through a column module and routes to a trigger column Tç T; then unfolds and generates h x w location-specific queries. These queries are i.i.d and can be parallelized across cores of gpu [48]. Each query TІ is passed through shared MLP and yields the vector fay and frigt MLP is queried iteratively
until whole grid f' comes into existence. Classification then involves comparing the averaged representation f with class-specific textual representations in the contrastive space (ii) Folded State: The parameters which the net learns are parameters of and MLP. (iii) Unfolded State: T expands yield h queries 'on-the-fly'. Learning involves oscillating this net between folded and unfolded states. This net can be trained via backpropogation 77 28
Despite the success of existing TTT approaches, several limitations need to be addressed [84 15 86] 1) The Information Bottleneck Problem (4 through many hidden layers multiple times, making it computationally expensive. 2) Reliance on surrogate pre-text task: the optimal data- data--uumennatioon pipeline the best pretext task not known beforehand, worsening the issue even further in an online setting. 3) Furthermore, TTT leverages architectures like transformers which rely on parallel perception: this requires projecting all input patches into shared representational space, thereby consuming significant memory.
Multiple TTT iterations requires foed-forward
Inspired by GLOM's philosophy [29], we hereby propose Asynchronous Perception Machine (APM), a new architecture for efficient cest-tmm-trainingg 1) It handles the bottteiinn-btttleeckk problem by directly learning a shorteut from input image to final representation from last layer of model (28 2) During TTT, we compute test-sample's s representation only once. Subsequent iterations involve over-fitting on this representation only and doesn't require any data- iitta-augmentttion//nnn task. 3) APM can operate on single patch at any location asynchronously [91] and still encode semantic-awareness in the net, thereby offering a fresh perspective towards machine- Seahhne-perception.
We make the following contributions in this work:
We propose APM, GLOM-inspired architecture that can perform test-time-training without
requiring data augmenttaiion/auxilaryyyy
APM is step towards validating GLOM's
philosophical insight: a percept is really a field (29]
APM is computationally efficient, i.e it almost halves the number of FLOPs over existing
methods [1 86). APM matches/surpasses TTT performance by 0.4%
8% across 16
datasets.
APM is architecturally simple: a convolutional layer and a single MLP of 5 layers. APM can still learn using one sample and semantically clusters on collection of images (28)
2 Background
We draw from insights previously philosophically mentioned in GLOM [29]. Consider how machine perception has been done classically: an input x E Rexxx transformed by a non-linear function
to a perceptual feature grid E RR xhxw c denotes image channels, h, w are input spatial dimensions.
Islands of agreement 29 Rather than viewing this matrix y as cuboidal grid, one can now see this as column vectors at each location. There are h x w such columns in y. Therefore, there exists a one-to-one mapping, between each input patch i.e. (x, v) with the vector U. that location. neural net [ can then thus learn to fit [I, agreement in GLOM (29). They were recently demonstrated on realistic datasets by (44, showing some promise that trained models can serve as free supervision-sourcee for vectors Uel
Ver These vectors have been termed islands of

---

Next, we imagine (eed-forwarding one patch (x, y) into the neural net f and estimating the response Vel It thus becomes possible to process these patches one-by-one rather than keeping them all together in memory 194] We then further imagine auto-regressively querying until the whole grid y has been brought into existence.
immpicct-erreeetaaaion/varraaar Fiiiid: The neural net f inputs a coordinate based query. (1, I, y) where (z, 3) is the 2D location in an image L. then gives the answer Vee [2]]. Such implicit representations have been studied extensively in 3D novel-view synthesis [60]. However, in the above problem I is a simple 2D image. In a recent work [29]. it was hypothesized that neural-ima iimplcctt representations can work on 2D images as well without having to retrain it every-time.
Layer Skipping [28]: We now combine the previous two insights together. Imagine that dhr has been estimated from a model after travelling through several layers. We can distill these ve into an implicit representation and learn direct mapping between input and last layer of model (28). During inference, we can skip forwaringg through all the model layers. The only feed-forward cost we then pay what it costs travel through f. Provided that the number of parameters in f are lesser than parameters in the teacher model, we can expect computational speedups.
We now introduce a connectionist-net f motivated by these insights. It can perform efficient test- time- tranntranning on a given 2D input.
3 Asynchronous Perception Machine (APM)
APM processes an input image I via two novel mechanisms: 1) A column module T which is said to contain an input image 7, 2) a column folding-unde unfolding mechanism that operates during each forward-pass pass. We first provide technical analogy to better understand APM.
3.1 A Technical Analogy 29
[2)
neural field does 3D novel view synthesis by querying an MLP with a point (zx, y, 2) and yielding corresponding rgb. In similar way, APM does 21 perception by querying an MLP with an image J, and location (ar, y) to yield location-specific feature fay. APM features a new mechanism to query the MLP, i.e column module T. Next, we define this column representation T.
3.2 Global Column Module: Defining compressed representation
We define column T as vector of dimensions 1 so that I' can summarize its entire identity. Given an image of dimensions e x h x w, we run a 2D convolution on it. Number of filters are set as accordingly. The resultant 1 x h w' feature map is then flattened into a single column vector T of dimensions d = ht x w' We shall refer to this column T "triggering hyper-column' (olummm((sed)))) (seed).. The only learnable parameters in this column are parameters of a convolution filter 92 91).
d. Our aim is map image 7 in this T.
3.3 Abstract view: The Column Infolding-Folding Mechanism
The trigger column T now starts undergoing cycles of bolding-unfolding (Fig TT. During unfolding, the column copies itself to yield h x w location-awen aware columns. During folding, these h x w columns collapse back into single column T. The neural-ne' then oscillates between these folded-une unfoldedd phases during learning iterations.
3.4 Computational Principle: Location-aware columns and their collapse
The unfolding-process shall now be concerned with generating location- column Tij from T. We generate h * w 2-D non-pann non-parametric positional encoding similar the ones being used in transformers [94) and neural fields [60] The trigger column Ti is then given by Ti = (T (T(p))) where denotes the stacking operator and (1, 1) S (i.j) S (H, W). T can said to encode identity
of an image.
The folding-prn proinns- involves collapsing all columns Ti back into T from which they had begun. This is achievable since the P.y used in Ti, was deterministic. An abstract-mathemaalee on folding is that all Pij in Tù get deleted/anmihilated at the end of every forward-pass, and only T is left. -ncoodingg contain periodic- sinussods which offers a strong positional-prion in
intuition
'The word triggering has been chosen because this column shall be used to trigger the queries to the neural field shared across different locations 29
3

---

practice 194]. Gradients collected from all Try then update the parameters of the convolutional filter the column T. This sharing of T across different locations (i now induces new fundamental
behavior 29 already hypothesized in GLOM (more details in section 5.3
This representation Tú (T)p exhibits strong-symmetry breaking behavior 29 For e.g.. consider different locations across same image 1. Although the identity T will be the same, the positional encoding Pij shall differentiate among different locations. The converse holds true: given the same position Pij in two different images, the column I shall differentiate among identities of different images.
All the experiments in this paper are performed withouutinjectiing the local patch Lay in the trigger column Tay. This showcases the myomm-yymmerry breaking behaviour of they can disentangle location-informalt from global -eiuiri-equuiiing an additional patch-prior.
3.5 Firing location-awart columns into shared MLP
Each column Tı is passed through an MLP to yield recation-specifii features fi, and RGB values RG Number of neurons in the first layer of the MIP same as dimensionality of column Tyy.
Column Independence: The MIP is shared across all the columns. One column is also independent of another as illustrated in Fig[] Therefore, the MIP can be queried sequentially. Firing a column Ty into the MLP yields a column vector Ve. Once h x w columns are finished firing, we get a feature grid of dimensions h X w c. Note that the number of columns can be as low as 1.
3.6 Training and Losses
We detail how the th iteration of rest-time-training could be performed. First, the obtained feature grid fxRaxuxe from APM is averaged to yield faug For the first TTT iteration, i.e. = 1, the image I is feedrrerd through multi-modal teacher like CLIP to get CLS token fas and corresponding text representation lats. APM then learns to estimate this same target feature sax. We enforce this by a simple L2 constraint as:
L L fele)
(1)
where faug,t is the averaged output feature from APM at a particular TTT iteration t. Note that the target fas is only estimated in f and remains the same for 2 2, i.e. subsequent feed-forward through teacher is not needed.
Memory-efficient estimation of faug,:: During a TTT iteration saog is computed as a simple average of fx This would require h x w columns to exist in the memory simultaneously. APM's s design assumes column independence which allows estimating faog as a statistical running average, i.e.
favg + fig n ++ assuming, n columns have already been fired into the APM and one additional column corresponding to position (i,j) of image is in the process of firing. This procedure is repeated until all positions (i,j) are exhausted. We represent the sequential column-firing firing by 'Gather-Grid' operator in Fig[
fang
Predicting image class-label: After certain TTT iterations, (say t = 20), the output feature faeg.t and the textual features tals are obtained. caage-classification then follows the standard practice of comparing the distance of faeg, with each plausible class feature tcls the contrastive space and choosing the closest one as the prediction [75].
Experimenting with APM
We quantitatively evaluate APM on zero-shot eest-tmmettraning on popular benchmarks containing distribution-shifts [5
86 84). Next, we quantitatively explore its computational efficiency.
contains level of corruptions on the test-set with 15 types of noises. Larger datasets with significant distribution shifts consists of ImageNet val/other curated splits. For e.g., ImageNet-Vas V2 contains natural images consisting of 10k images across 1000 classes. ImageNet- A contains 7500 adversarial-image consisting of 200 categories. ImageNet-R consists of 30000
Datasets: Cifar-10C
86
'It means generating a unique representation Ti, for any image and a location selected in it. Another way to break symmetry is to add noise [40). Noise also helps escape degenerate mealmmminima. These ideas can be traced back to bottanan--achhiee//hhooe machines/hoptield-nes 44.

---

artistic-image across 200 ImageNet categories. ImageNNt--ketch consists of black/white sketches of 50000 images for 1000 classes. ImageNet-C consisting of 15 types of corruptions with 5 levels of severity. There are additional 9 cross-dataset generalization datasets
84
Baselines: We compare against standard TTT-online [86]. TTT-MAE 13 TPT 84 CLIP VIT-B/16,
Coop, CocoOP. We also benchmark CLIP VIT-L/14 and the strongest OpenCLIP VIT-H/I44quickgellu variant pre-trained on dffn5b.
Table 1: APM's Robustness Natural Distribution Shifts. CoOp and CoCoOp are tuned on
ImageNet using 16-shot training data per category. Baseline CLIP, prompt ensemble, TPT and our APM do not require training data. A / in means that method leveraged pre-trained weights on clean variant of train set aka, Image-net net and downstream-tu on corrupted version.
Method
ImageNet Topi
ImageNot-A
ImageNot-V2 Topl
ImageNet-R Topl acc.
ImageNot-Sketch Topl
Average
OOD Average
Topl
CLIP VIT-/6
K
66.7
47,8
60.8
73.9
46.0
59.1
57.2
Ensemble
61.8
61.2
59.4
TPT
X
68.9
54.7
63.4
77.0
47.9
62.4
60.8
APM (Ours)
X
68.1
52.1
67.2
76.5
49.3
62.6
61.2
CoOp
71.5
49.7
64.2
75.2
47.9
61.7
59.2
CoCoOp
7L.
64.0
62.1
59.9
TPT + CoOp
73.6
57.9
66.8
77.2
49.2
64.9
62.8
TPT CaCoOp
62.6
CLIP VIT-1/14
K
76.2
69.6
72.1
85.9
58.8
72.5
71.6
APM (Ours)
62.2
74.2
73.4
OpenCLIP-VTTHHH/
X
81.6
79.1
80.7
92.9
72.8
81.4
81.3
APM (Ours)
X
84.6
84.2
83,9
94,9
77.1
84.9
85.0
Table 2: APM's performance on ImageNet-C, level 5. The first three rows are fixed models without test-time training. The third row, ViT probing, is the baseline used in 15] A // in P means that method leveraged pre-trained weights on clean variant of train set aka, Image-net and downstream-tu on corrupted version. CLIP VIT-L/14 is generally more robust. APM does better on 11/15 noises
with an average accuracy score of 50.3.
brigh
cont
defoc
elost
Fog
Frost
gains
glass
impul
jpeg
moin
pixel
shot
snow
200m
Averge
Joint Train
62:3
4.5
26.7
39.9
25.7
30.0
5.8
16.3
5.8
45.3
30.9
45.9
7.1
25.1
31.8
24.8
Tune
67.5
7.8
33.9
32.4
26.4
38.2
22.0
15.7
23.9
51.2
37.4
51.9
23.7
37.5
37.1
33.7
VIT Probe
68.3
6.1
24.2
31.6
38.6
38.4
17.4
18.4
18.2
51.2
32.2
19.7
18.2
35.9
32.2
24.2
TTT-MAE
98
63.0
41.9
OpenCLIP
71.9
47.0
32.7
46.9
26.0
28.1
62.7
37.7
28.2
50.4
(Ous)
77.4
51.9
56.6
37.9
64.8
53.2
28.7
31.4
33.0
68.4
44.1
64.5
33.1
56.9
43.9
50.3
Results and Analysis: We study APM's performance on test-time-trainiie on several datasets. APM processes each test sample individually: i.e. the weights are drawn from a normal distribution after processing every sample prevent information leakage. For zero-shot classification of a test-sample, APM leverages the 80 textual-prompl templates similar to the ones used in CLIP. In Tab T. APM scales up to zero-shot classification task datasets with 1000 classes. Using CLIP-ViL ViT B/16 as a teacher, we surpass TPT (84) with an avg score of 62.6 and avg ood-score of 61.2. Next, we benchmark OpenCLIP-VITH/LI against all these splits. Using the same model as our teacher, we get absolute improvement of 3% t [84.6%] on ImageNet val set, 5.1% [84.2%] on ImageNet-A, 3.2% [83.9%] respectively. A similar trend is observed with VIT-L/14 backbone. This might lead to the conclusion that stronger teacher seems to benefit APM. In Table 2 we show results on Imagenet-C. Using a CLIP VIT-L/14 teacher, APM gets highest accuracy 11 11/15 noises with the highest average score of 50.3. Note that TTT-MAE also uses VIT-L encoder, and is prelruined. In contrast, APM doesn't need any dataset ppecific-pretraining Finally, in Tab 3 APM improves upon 4/9 datasets, comes close on remaining 5 and gets the highest average accuracy score of 65.5.
on ImageNet- 22, 2% t [94.9%] on ImageNet-R R, 4.3% [77.1%] on ImageNet-
APM is computationally elficient: All experiments are run on a same desktop- worksttioo containing 1x rtx a6000/96GB ram/Ubuntu 22.04/2TB ssd. Flops were counted with meta's fvcore library 889. In Tab4. we perform 20 TTT iterations. TTT on CLIP-VIT B/16 baseline used in 84 consumes 462Gflops. Next, we do TTT leveraging APM. At 1, feed-forwards involves clip-teacher and consumes 20.5 flops. Remaining 19 TTT iterations only involve overfitting on distilled image token at / = 1, and consumes 10 flops/TTT-ttrationn.. The entire profile-dump yields 241.7 flops, which is almost 50% reduction over 462 flops. The total memory occupied by APM i.e. B.7GBt more than CLIP-VIT/B 2.3G since teacher kept in memory during TTT. However, APM only occupies 600 B and reduces actual consumed-fion by 50%.
5

---

Table 3: Cross-datased generalization from ImageNet to fine-grained classification datasets. CoOp and CoCoOp are tuned on ImageNet using 16-shot training data per category. Baseline CLIP, prompt ensemble, TPT and APM do not require training data or annotations. We report top-1 accuracy.
Method
P Flower102
DTD
Pets
UCF101
Caltech101
Food101
SUN197
Aireraft
FumSAT
Average
CoOp
68.7
41.9
89.1
66.5
93.7
85.3
64.2
18.5
46.4
63.9
CoCoOp
/
70.9
45.5
90.5
68.4
93.8
84.0
66.9
22.3
39.2
64.6
CLIP-VVT-B/16
Ensemble
X
67.0
45.0
86.9
65.2
93.6
82.9
65.6
23.2
50.4
64,6
IPT
X 69.0
47.8
87.8
68.0
94.2
84.7
65.5
24.8
42.4
APM (Ours)
X
62.0
48.9
81.6
72.6
89.6
84.2
65.7
29.7
55.7
65.5
Table 4: APM's computational analysis: TTT for 20 iterations on APM. Baseline is CLIP VIT-B/16
which is used teacher in 84 Monas (GB), Gelopsmeas (GB) tmeasured stats. M, G.Flops are idealistic estimates. (t): tth ttt iteration, (s): student, (u): teacher, (s+u): portion of memory/flops consumed by student/teacher respectively. Note that APM is a 25 net.
t
Params(M)) (GD)
MAGB)
GFlops
GFlops,
Swin
1-20
87
1.5
1.4
353
308
TPT
1-20
151.3
2.7
476
CLIP VIT-B/16
1-20
149.2
2.3
1.8
462
410
CLIP VIT-B/16(U)85
1
149.2
1.8
1.8
20.5
20.5
APM(s)
1
174.2(s*u)
2.7 (s+u)
1.8()) 0.6(s)
20.5(u)
20.5(u)
APM(s)
2
174.2(s+l)
1.8(u)+ 0.6(8)
10(s)
Peak Use
1-20
174.2(s+u)
2.7 (5+u)
18(u) 0.6(s)
241.7 (s+u)
210.5 (s+u)
GigaFlops vs No of patches
Inference Time vs No of Patches
CLIP BIT-
2
CLIP (IT-B/16(total)
FigpFlops
19
1.5
14
AP (total)
Time
0.5
APM (Gilops/it)
APM
14
28
56
112
224
28
112
224
Thpatch
Tbyatch
Figure 2: APM's analysis with variable number of patches: (left) Gilops of CLIP VIT-B/16 and APM function of number of processed patches. (right) Feed-forward time vs number of patches.
APM does patch-based processing: In Fig 2 we estimate GFlops/time to process a 224 224 image using CLIP VIT-B/16 & APM. CLIP VIT-B/16 consumes 20.5 Gflops On the other hand, APM takes only 10 Gflops in total, in part due to lesser parameters. APM's effectiveness comes from processing as low as patches at the same time: it occupies lesser memory but takes more time (i.c. 1.5 seconds). 1.5sec is still lower than VIT-B/16's 2. 2.2sec. The extreme lies when all patches processed: Inference time in APM goes down to 0.002 seconds compared to VIT-B/16's 2.2 seconds, thereby indicating its effectiveness. Note that VIT-B/16 can't do this patch patch-based processing.
APM's computational effectiveness stems from this unique ability overfit on a single test sample's embedding which was distilled only at = 1. This merits deeper investigation.
Whole
Part
Input
Iter
1e-3 Iter 50
Iter 100
Iter 150
Iter 200
Iter 250
1e-12
Figure 3: Overfitting single distilled token representation leads to islands of agreement[29] APM is overfit on a test-sample's representation distilled from a teacher. We plot t-sne clustering of output features over 250ttt iterations. La loss between predicted features and distilled sample falls from le-3 to le-12. 12. Moving left to right shows that wholes break into smaller parts.
APM can learn on single sample: In Fig 3 feed-forward single image through our APM and perform 250 TTT iterations We only overfil on one distilled test-sample embedding and plot the 2D
A larger number of TTT iterations are needed to perceive semantically meaninggfu islands. Actual TTT process is far quicker, it takes around 20 iterations.

---

t-sne clustering of predicted APM. It can be seen that the elements of the scene have gradually started cluster. This suggests that APM solves an inverse problem: given a single test-sample embedding. what features in the image led to its formation? Over several TTT iterations, APM's features become representative enough to explain different parts of a scene Therefore, the same-net could be made to move up-down the part-whole hierarchy 288 although it requires TTT-iterations for now.
Next, we explore if data augmentation improves APM's performance. We perform TTT iterations on CIFAR-10' 10's test set and find that data augmentation drops APM's performance from 98.6 to 76.7. This quantitatively demonstrates that APM works best when it does vae-saaple overfitting. It now aligns with the earlier qualitative experiment: over TTT-ierationss, the network is learning to cluster the elements in the scend3] Data augmentation distorts the sample and makes it difficult for APM's predicted features to agree on a stable, relaxed- represstttt that explains the scene 42 26
Till now, APM's operation has involved random-initializatio of weights for every test-sample and performing test-time-rraining. There is another mode that it can be made to operate in.
5 APM Training (Qualitative Analysis)
APM can also scale up and do learning on a batch of samples (for e.g.. COCO images (50) distilled from a teacher[31] [.]. This requires introducing several new mechanisms. Note that this section is meant to qualitatively demonstrate how scaling up APM can improve the net's interpretability, and help seed future research. Quantitative experiments beyond test-time- pest-tme-traiinin remain out of the scope of this paper. APM's training follows a standard setup in suusserrieed-eerrnnnng the full algorithm for ASL--traning//estt-ime--- -rriiiiig in the Appendix[] During inference, APM takes any 2D image I& and predicts its RGB reconstruction RG Bk/higher dimensional features fi. The net then begins to demonstrate several interesting properties, which we will discuss next.
We have provided
5.1
APM can do RGB reconstruction for any 2D input.
RGB ml
fout
Q
Folded
T
APM
Attention
L
CLS
Unfolded
T: Tiy
RGB Head
RGB:,
Figure 4: RGB Decoding in APM: Input trigger column Tij is concatenated with predicted feature fiy and fed to downstream RGB head. This decodes RGB logit at location (i.j) for any 2D input Tg. (ii) Input 24 sampled from Coco-val set. RG reconstructed RGB, fout: Predicted feature grid.
Given a sample Iı. APM can reconstruct its RGB. In Fig4 we achieve this by estimating frgh, (Tylfty). ). This skip-connectionn from trigger column Ti to output feature has a subtle reason: consider a white dog and brown dog. The predicted object-level feature for both will be almost identical 22 2 However, Tú is different for both since it contains lower patch-level features (29]. Therefore, this helps us break symmetry. Without this skip-connection (2). the net fails to predict RGB. The network is trained to reconstruct RGB for a batch of images, Lygb = >N E/ Le(p),,,;). Pj). where py/p; are ground truth/predicted RGB-logits respectively.
5.2
APM is asynchronous yet encodes semantic-awarenes in the net.
Given a sample Tk. APM can directly learn to mimic the entire last layer feature- -rid which a teacher model would have generated. We enforce this by Lggrd = = Ej=1
Neww I (2 fyrid) In Fig B we estimate the eтror map between the features predicted by our APM and 667. The error map is mostly black which shows that it closely approximates Dinov2's grid Note that APM does patch-basee based asynchronous processing whereas DINOv2 relies on parallel perception. Finally, Fig Siii) shows a simple feed-forward of CIFAR-10 sample through APM. We can see semantically-an awaare features. Note that this is single feed-forward through APM Predicting output feature grid allows the net to encode dark knowledge, i.e. the knowledge of both correct.
TThis particular net got La grid loss of 0.1
7

---

Input
Dinov2
APM
Error Map
(i TTT
(ii) SSL-Trained
(iii) Inference
Figure 5: APM feature Analysis: (i) TTT iterations
input image leads semantically aware
clustering. top: 2D t-sNE. bottom: 3D t-sNE. 64 [2]. (ii) APM is trained via self-supervision using DINOv2-Teacher. (from left) Input, Dinov2 grid, APM grid. APM's grid closely approximates Dinov2 grid evident from black regions in error map. Note that APM does asynchronous patch- processing whereas Dinov2 does parallel perception. (iii) Cifar-10 samples feed-forwarded through SSL-trained APM yields features significant semantic quality. (29
and incorrect classes 30). This is better than just mimicking one hot vector target class since manages encode lower probabilities of wrong classes also.
5.3 APM is step towards validating GLOM's insight: input percept is field field]36] 36
Source
5
Target
Figure 6: APM is step towards validating GLOM's insight [29]: input percept is field. An interpolation between any two images in the wild. This field arises APM's MLP consisting of layers. Trigger column T acts key which retrieves image from the APM's memory. T resides continuous embedding space, not discrete addressing space.
In Fig 6 we show that APM can interpolate between any two images the wild. We choose two images 11 and 12. These images are then funneled through the trigger column T and yield two vectors Un and, 1/ respectively. Next, we generate n intermediate latents separated by an equal linear distance by vi 01 -2-21 Each latent then brings into existence its own set of location- and decodes an image from the MLP. Such an interpolation has been previously observed other models (20). APM now functions new form of addressing mechanism: the trigger column T acts a key. Copying across locations yields image-specing queries [29]. Values are synapses triggered in the MLP. RGB decoding happens in the output head. Hence, such continnous keys and queries exist outside the net
columns
Classically, auto-regression has unrolled shared-decoded over time 98 In contrast, APM holds the whole sequence in T, and directly hops onto space/time-step 12 by querying the MI.P with docation-column Tij. Note that Ti is generated by unfolding T. Recurrenceffeedback-tno
are compensated for by form of feature-expression [29]. This step towards validating GLOM's insight, i.e. input-percept is [29] and one can now interpolate in it (gestalt psychology). Furthermore, the trigger column T resides a continous embedding space, and not discrete hardware locations(classica Al) 29). Therefore, APM tries to integrate insights from both fields.
Ablations on APM
The experiments on TTT had relied on curious ability of APM: it could simply overfit on test-sample's distilled representation f 1. This merits further investigation:
Effect of one-sample test-time training: In Table 5a we investigate whether existing networks are capable of one-sample cest-imme-trainng We employ a handomly-initialize network to overfit on a distilled test sample's token obtained in the first TTT iteration (28). Standard MLP achieves low accuracies of 9.0 and 3.8 on CIFAR-10 100 and CIFAR- 1000 respectively. Notably, an 1 11.4M parameter ResNet18 outperforms the larger ResNet34 with 21.5M parameters. A reason might be that too many parameters in ResNet34 gives it too many degrees of freedom 32 it finds it hard to overfit one

---

Table 5: Ablations on APM. All nets except CLIP VIT-L/14 use random weights b) To: trigger column contains convolutions. Tw Trigger column contains routed VIT representation. C-10: CIFAR-10, 10, C-100: CIFAR-100. Accuracy is reported.
(a) Ablation evaluate abilities existing nets to learn from a single sample 22
(b) Ablations our APM on C-10.lay means that local patch was injected into the column Te.
Params
C-10
C-100
Lirrid
Tecls
Legb
Tzy
Acc
Zeroshot
Te
X
X
X
94,2
CLIP VIT-L/14
428M
95.37
73.28
Te
/
X
91.0
MLP
21M
9.0
3.8
T
/
X
X
96.1
ResNet18
11.4M
85.69
21.77
T.
/
/
J
X
96.5
ResNet34
21.5M
78.24
12.89
T.
/
V
/
96.8
APM
25M
97.04
77.98
Tust
98.6
sample. Our APM demonstrates strong performance on both CIFAR-10 and CIFAR- 100, surpassing the CLIP VIT-L/14 baseline.
Effect of various losses on APM: We analyze each row in Table 5b Initially, only the CLS token from the teacher was distilled into our network, resulting in a yield of 94.2% 94.2%. When we added RGB reconstruction loss for input, accuracy dropped to 91.0, attributed to the difficulty of breaking RGB symmetry [23]. Subsequently, mimicking the entire feature grid and CLS token from the teacher increased accuracy to 96.1. Adding both Ligh and Lgrid further improved performance to 96.5. Legria here refers to the last-layer of the teacher. Notably, while simple RGB reduction decreased performance (91.0), combination of Lgrid and Lygb enhanced our network, as lower RGB and higher object features complement each other [29]. Injecting local patch information Lgy into the trigger column improved performance to 96.8. Finally, routing output tokens from single VIT layer into the trigger column Tost strengthened the column and improved performance.
Effect of increasing number of convolution in T: Increasing number of convolutional kernels from 1 to 3 improves from 96.08 to 97.67.
These ablations reveal: 1) APM can do one-sample overfitting for a test sample, 2) It helps to have both local patch and a strong image representation in the trigger column T, and 3) Increasing the levels of part-whole supervision strengthens the net.
7 Related Work
Prompting is a mechanism to adapt a foundational-mode downstream task in zero-shot manner[58). However, prompting typically requires well-designed hand-crafted prompts. Prompt-tuning methods consist of learnable prompts which enable a parameter-efficien approach to fine-tune
Prompting Approaches:
iondataionalmooder CoOp[I06] applied prompt-tunin ponpt-tuuing to CLIP. However, Co6p[I06] is sensitive to OOD-data, which CoCoOp in turn compensates for by conditioning the prompts on model inputs. Similarly, TPT[84] optimizes a prompt to encourage consistent predictions across multiple augmented views of the same test sample, and uses confidence-selectionn to filter out noisy-pu sriisy-redictions. Note that TPT [84) performs prompt-tuning over VTT-B/16 but requires foed-forward through all the model-layers for every itcration.
Best-Time-Ge Oatnne--timizziion Introduces the notion where a model adjusts its decision boundary dy- namically during testing, for eg, improving robustness to distribution shifts. Test-time training (TTT) generally adds self-supervised multi-task branch, and performs a SSL-task like rotation, or masked- reconstruction adapt the network to the jest-sample[[S] [1]. These approaches typically initialize the net with pre-trained weights, for eg, Imagenet before undergoing utt-iterations on a downstream corrupted-datased An alternate line of work, for eg, TENT[96] proposes to minimize entropy of batch-wise test-samples. However, TENT[96] requires more than one test-sample to converge towards an optimal solution, whereas APM can also operate on one test sample. Another line of work adjusts internal aatch-norm-sta of network [8]] However, this makes the aetwork-architectur inflexible and requires more than one test sample for optimization. Several other papers following the original pioncering- paper [86) have worked on different problem formulations, for eg, assuming access
to an entire dataset (e.g. [51 72 103
18
104
or a batch (e.g. TENT (96) of test inputs.
In contrast, APM evaluates on each test-sample independently. Inductively. APM does not re- quire any dataset specific pre-training, a pretext task or prompt tuning. Some approaches also use higher-parane parreeeteraeeeee transformer/diffusiion models (72 making the optimization compute-intensive. However in APM, for TTT iteration / 1, feed-forward is done through only 25M parameter APM and not the 149.2M Clip VIT-B/16, resulting in computational efficient wew-ttmmtttrrinnge 2

---

APM also inherits zero-shot behaviour from CLIP, which allows it to bypass training a separate dataset-specific aheaa-ppoobe for downstream TTT. Finally, some works areas like source-free domain-adaptation do perform TTT on smaller datasets like Cifar-10 etc 196.. APM additionally shows results on Imagenet splits [2 and various cross-generalizattonn datasets(Tah(3)
The idea of encoding part-whole dynamic-parse-trees as distributed rep- resentations in neural nets can be traced back to [33] with recent attempts leading to capsule networks 42 [79]. However, the 2MM42]/attentiion-baaeee (7] routing forces each capsule to rep- resent only one part 64
Part-whole hierarchies:
62 29 This fundamental flaw prevents capsules from scaling-up and generalizing to multiple OOD objects[644 Recently, GLOM proposed theoretical system of representing each input pixel as containing a column-vecton These vectors then undergo routing procedure such that pixels corresponding to same part come to "agree' with each other. [T]] demon- strated GI GLOM on Cifar splits. APM does not need any routing because processes location-aware columns independently. APM now reveals an additional earlier(Figā).
property not shown
Knowledge Distillation: There has been a long history of training data-specific mixture/product of experts and having their ensemble vote towards prediction, with the key idea make the experts as different from as each other as possible, and only one expert deciding on the predictions of a particular imput sample 4 This is better than having all experts give equal opinion on sample 34] Other methods attempted to 'gather' their collective knowledge to a single model for edge-device deployment 31] Recently, knowledge has been transferred from larger teacher model to smaller ones 5. for eg, foundational model [8 74. The student often retains/eccomes-beeee
than its teacher 1 In practice, the weights of teachers are fluctuated slowly (EMA) as compared to the student 87 This mechanism then helps realize Kahnman's 's theory of slow-fast thinking (7).
Typically, distillation requires poltzmann-matching
predicted distribution between students and teachers, with the distribution 's sharpness being governed by a temperature parameter [40). In contrast, APM directly mimics the entire last layer feature grid of a teacher via Le norm 993] Furthermore, APM possesses a -ovel-induetive bias that can recover comantic-featuree from a single CLS-toker distilled from a teacher(Fig(5]. This validates the intuition that CLS tokens encode useful geometric- information of a scene after cross- cooss-attenioon of CLS token with patch tokens of an input image
Routing Mechanisms: Routing mechanisms involve routing correct object-specific information to correct neurons. [54) proposed slots which perform binding by iterative rounds of self-attention. Löwe A1(5 proposed 2D complex autoencoder and rotating (57tures[57] where presence of object is encoded in phase of a neuron and follows the minimum description length principle(39) In APM, binding done via the location 22s) GroupVIT [IOI] routes information multiple group tokens and shows that semantic segmentation emerges with just image-text contrastive supervision.
8 Conclusion
We propose APM, inspired from the insights presented in 29 38
APM promises to APM
be an efficient architecture for test-iimetrrainng and asynchronous shows robustness to extreme distribution shifts 24]. APM demonstrates that MLP's can be made to semantically cluster given image. We hope that APM will help inspire further research on simpler weight-sharing, lower-memory, nigher-bandwiid efficient-nets (28).
Limitations: In this work, we have mainly-focused on image-llassification Furthermore, APM requires multiple TTT iterations for now, although they might be reduced by exploring pre-training on a source-dataset (88) APM can work on just I sample with randomly -intomyy-iiiializedd weights. However, it still requires a single CLS token which has been distilled from a teacher pre-trained on a large-scale dataset. Our eesliinnary-xpprimeeeee
have revealed that APM can still do without a teacher 23), showing potential that APM can be self-sufficient and independent.
Acknowledgements
This research is supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/ Interior Business Center (DOI/BBC contract number 140D0423C0074, The U.S. Government is authorized to reproduce and distribute reprints for Govermmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official
10

---

policies endorsements, either expressed or implied, of IARPA, DOI/BC, or the U.S. Government. We also thank many other people and MLCollective whose support made this work possible.
References
[1] Shahzad Ahmad, Sukalpa Chanda, and Yogesh S Rawat. Ez-clip: Efficient zeroshot video action recognition. arXiv preprint arXiv:2312.08010, 2023.
[2] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. arXiv preprint arXiv:21 2(3):4, 2021.
[3] Yoshua Bengio. The consciousness prior. arXiv preprint arXiv: 709900868, 2017.
[4] Ludwig Boltzmann. Lectures on gas theory. Univ of California Press, 2022.
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-ers object detection with transformers. European conference on computer vision, pages 213-229 Springer, 2020.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 9650-9660, 2021.
[7] Carlo Collodi, Shirley Goulden, et al. The Adventures of Pinocchio. Lancer Books, 1968.
[8] Francis Crick and Graeme Mitchison. The function of dream sleep. Nature, 304(5922):
111-114, 1983.
[9] Charles Darwin. On the origin of species, 1859, 2016.
[10] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.1 1929, 2020.
[12] Albert Einstein. On the electrodynamics of moving bodies. 1905.
[13] Albert Einstein. Quantentheorie des einatomigen idealen gases. zweite abhandlung. Albert Einstein: Skademie-Vortrüger Sitzungsberichte der Preußischen Akademie der Wissenschaften 1914-1932, pages 245-252 257, 2005.
[14] Brendan Frey, Peter Dayan, and Geoffrey E Hinton. A simple algorithm that discovers efficient perceptual codes. Computational and Psychophysical Mechanisms of Visual Coding. Cambridge University Press, Cambridge, UK, 1997.
[15] Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked autoencoders. Advances in Neural Information Processing Systems, 35:29374-29385, 2022.
[16] Nicola Garau, Niccoló Bisagno, Zeno Sambugaro, and Nicola Conci. Interpretable part-whole hierarchies and conceptual-semane relationships in neural networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 13689-13698, 2022.
[17] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 456. JMLR Workshop and Conference Proceedings, 2010.
[18] Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lee. Note: Robust continual test-time adaptation against temporal correlation. Advances Neural Information Processing Systems, 01227223-272666 2022.

---

[19] Taesik Gong, Yewon Kim, Taeckyung Lee, Sorn Chottananurak, and Sung-Ju Lee. Sotta: Robust test-time adaptation on noisy data streams. Advances in Neural Information Processing Systems, 36, 2024.
[20] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11))139-144 2020.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings the IEEE conference on computer vision and pattern recognition. pages 770-778, 2016.
[23] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[24] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint 1903.1:903..226 2019.
[25] Peter Robin Hiesinger. The self- brain: how neural networks grow smarter. 2021.
[26] G Hinton. Using relaxation to find puppet. In Proceedings of the 2nd Summer Conference on Artificial Intelligence and Simulation of Behaviour, pages 148-157, 1976.
[27] Geoffrey Hinton. Where do features come from? Cognitive science, 38(6):0078-110)) 2014.
[28] Geoffrey Hinton. The forward-forwar algorithm: Some preliminary investigations. arXiv preprint arXiv:2212.13345, 2022.
[29] Geoffrey Hinton. How to represent part-whole hierarchies in a neural network. Neural Computation, 35(3):413-452 2023.
[30] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Dark knowledge. Presented the keynote in BayLearn, 2(2), 2014.
[31] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge a neural network. arXiv preprint (XXi:11503.02311,, 2015.
[32] Geoffrey E Hinton. The horizontal-vertiee delusion. Perception, 16(5):677-680) 1987.
[33] Geoffrey E Hinton. Mapping whot- hierarchies into connectionist networks. Artificial Intelligence, 46(1-2):47-75, 1990.
[34] Geoffrey E Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 14(8):1771-1800 2002.
[35] Geoffrey E Hinton. To recognize shapes, first learn to generate images. Progress in brain research, 165:535-547, 2007.
[36] Geoffrey E Hinton and James A Anderson. Parallel models of associative memory: updated edition. Psychology press, 2014.
[37] Geoffrey E Hinton and Zoubin Ghahramani. Generative models for discovering sparse distributed representations. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 352(1358):1177-11999 1997.
[38] Geoffrey E Hinton and James McClelland, Learning representations by recirculation. In Neural information processing systems, 1987.
[39] Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference on Computational learning theory, pages 5-13, 1993.
12

---

[40] Geoffrey E Hinton, Terrence J Sejnowski, and David H Ackley, Boltzmann machines: Con- straint satisfaction networks that learn. Carnegie-Mellon University, Department of Computer Science Pittsburgh, PA, 1984.
[41] Geoffrey E Hinton, Peter Dayan, Brendan J Frey, and Radford M Neal. The" wake-sleep" algorithm for unsupervised neural networks. Science, 268(5214): 1111111111111 1161, 1995.
[42] Geoffrey Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. In International conference on learning representations, 2018.
[43] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances neural information processing systems, 53:6840-6851, 2020.
[44] John Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 79(8)22554 2558, 1982.
[45] Edwin Hubble. The evolution of the universe. Proceedings of the National Academy of Sciences, 1563):168-173, 1929.
[46] Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
[47] Daniel Kahneman. Thinking fast and slow. Farrar, Strauss and Giroux, 2011.
[48] Alex Krizhevsky, One weird trick for parallelizing convolutional neural networks. arXiv preprint 1rXiv:140..5997, 2014.
[49] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle- adjusting neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5741-5751, 2021.
[50] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer
2014: 13th European Conference. Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[51] Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. Tit++: When does superviue test-time training fail or thrive? Advances in Neural Information Processing Systems, 34:21
2021.
[52] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
[53] Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljašić, Thomas Y Hou, and Max Tegmark. Kan: Kolmogorov- networks. arXiv preprint arXiv:2404. 19756, 2024.
[54] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33: 13:11153 11211-151111111 1538, 2020.
[55] Sindy Löwe, Peter O'Connor, and Bastiaan Veeling. Putting end to end-to-end: Gradient- isolated learning of representations. Advances in neural information processing systems, 32, 2019.
[56] Sindy Löwe, Phillip Lippe, Maja Rudolph, and Max Welling, Complex-valued autoencoders for object discovery, arXiv preprint arXiv:2204.02000 02075, 2022.
[57] Sindy Löwe, Phillip Lippe, Francesco Locatello, and Max Welling. Rotating features for object discovery. Advances in Neural Information Processing Systems, 36, 2024.
[58] Neelu Madan, Andreas Møgelmose, Rajat Modi, Yogesh S Rawat, and Thomas B Moeslund. Foundation models for video understanding: survey. arXiv preprint arXiv:2405.037700 2024.
13

---

[59] Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36:53038-53015 2023.
[60] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, (5(1):99-106) 2021.
[61] Rajat Modi. Detection with r-cnn, 2020. URL 05/17//detectionrrcmm//
10/2020/
[62] Rajat Modi, Naman Garg, and Harish Parthasarathy. Steganography using wavelets with statistical performance analysis. In 2017 1st International Conference on Electronics, Materials Engineering and Nano- Nano-Technology (IEMENTech), pages 1-8, IEEE, 2017.
[63] Rajat Modi, Aayush Jung Rana, Akash Kumar, Praveen Tirupattur, Shruti Vyas, Yogesh Singh Rawat, and Mubarak Shah. Video action detection: Analysing limitations and challenges. arXiv preprint arXiv:2204.07885 2022.
[64] Rajat Modi, Vibhav Vineet, and Yogesh S Rawat. On occlusions in video action detection: Benchmark datasets and training recipes. In seirth--eventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[65] Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks. Google research blog, 20(14):5, 2015.
[66] J von Neumann. Theory of self-reproducing automata. Edited by Arthur W. Burks, 1966.
[67] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:230..07193, 2023.
[68] Roger Penrose. The emperor's new mind: Concerning computers, minds, and the laws of physics. Oxford Paperbacks, 1999.
[69] Roger Penrose. Before the big bang: an outrageous new perspective and its implications for particle physics. Proceedings of EPAC, pages 9759-2763, 2006.
[70] Roger Penrose, Stuart R Hameroff, and Subhash Kak. Consciousness and the universe: Quantum physics, evolution, brain & mind. Cosmology science Publishers., 2017.
[71] Stefano Piraino, Ferdinando Boero, Brigitte Aeschbach, and Volker Schmid. Reversing the life cycle: medusae transforming into polyps and cell transdifferentiation in turritopsis nutricula (cnidaria, hydrozoa). The Biological Bulletin, 190(3)302--312, 1996.
[72] Mihir Prabhudesai, Tsung-Wei Ke, Alexander Cong Li, Deepak Pathak, and Katerina Fragki- adaki. Diffusion-tta: Test-time adaptation of discriminative models via generative feedback. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023.
[73] AC Bhaktivedanta Swami Prabhupada and Bhaktivedanta Swami. Bhagavad- as it is, Chapter 8, Verse 18-19, Bhaktivedanta Book Trust Los Angeles, 1972.
[74] Biqing Qi. Xinquan Chen, Junqi Gao. Dong Li, Jianxing Liu, Ligang Wu, and Bowen Zhou. Interactive continual learning: Fast and slow thinking. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12882 12892, 2024.
[75] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8763, PMLR, 2021.
[76] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. Proceedings of the IEEF/VV conference computer vision and pattern recognition, pages 10684,1069,, 2022.
14

---

[77] Frank Rosenblatt. The perceptron: probabilistic model for information storage and organiza- tion in the brain. Psychological review, 65(6):386, 1958.
[78] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by pack-propag errors. nature, 323(6088):533-536 1986.
[79] Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. Advances in neural information processing systems, 30, 2017.
[80] Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models mirage? Advances in Neural Information Processing Systems, 36, 2024.
[81] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adapta- tion. Advances in neural information processing systems, 33:1 1539-11551, 2020.
[82] Terrence J Sejnowski, Paul K Kienker, and Geoffrey E Hinton. Learning symmetry groups with hidden units: Beyond the perceptron. Physica D: Nonlinear Phenomena, 22(1-3):260-25 1986.
[83] Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-423, 1948.
[84] Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision- llnguuge models. Advances in Neural Information Processing Systems, 11114244-4228, 2022.
[85] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning, pages 9228- PMLR, 2020.
[86] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International conference machine learning, pages 9229-9248. PMLR, 2020.
[87] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averagee con- sistency targets improve supervuper deep learning results. Advances in neural information processing systems, 30, 2017.
[88] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, fean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 2023.
[89] The FAIR Computer Vision Team. fvcore: A wight core library for common computer vision functionality. htt.ocoiiiyttttse///tttt.ooo//oe http:///tccccccccccceeoeeeeereee
2024.
[90] Alan M Turing. Computing machinery and intelligence. Springer, 2009.
[91] Alan Mathison Turing. The chemical basis of morphogenesis. Bulletin of matheraticaa biology, 157- 1990.
[92] Alan Mathison Turing et al. On computable numbers, with an application to the entschei- dungsproblem. J. of Math, 58(345-363):5, 1936.
[93] Haithem Turki, Jason Y Zhang, Francesco Ferroni, and Deva Ramanan. Suds: Scalable urban dynamic scenes. In Proceedings of the IEEE/VVF Conference on Computer Vision and Pattern Recognition, pages 12375-12385, 2023.
[94] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[95] John Von Neumann, Arthur Walter Burks, et al. Theory of self-reproducing automata. 1966.
15

---

[96] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006. 2020.
[97] Qin Wang, Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7201-7211, 2022.
[98] Paul J Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78(10):1550-1500 1990.
[99] Stephen Wolfram et al. A new kind of science, volume 5. Wolfram media Champaign, IL, 2002.
[100] Junfeng Wu, Qihao Liu, Yi Jiang, Song Bai, Alan Yuille, and Xiang Bai. In defense of online models for video instance segmentation. In European Conference on Computer Vision, pages 588 605. Springer, 2022.
[101] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18134-18144, 2022.
[102] Hong-Xing Yu, Leonidas Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. arXiv preprini arXiv:2107.07905, 2021.
[103] Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15922-15932, 2023.
[104] Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In Proceedingy of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 15922-15932, 2023.
[105] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu, Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 16816-16825 2022.
[106] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2377-2348, 2022.
16

---

Appendix: Asynchronous Perception Machine for Efficient Test-Time Training
A Broader Impact
There are two main ideas that led us to Asynchronous Perception Machine [40 22 is that instead of thinking of features as a cuboidal feature grid [2], one can think of it as a column vector at each location 29 64.. This helped us learn one to one mapping between the input rgb patch and the vector at particular location. It led to the net being able to process one patch at time.
The first idea
The second idea is this notion of collapsing information into single starting point. Our previous understanding was that this 'collapse' leads to degeneracy [40]. In this paper, the information can be recovered from the starting point by copying it many times and breaking symmetry with positional encoding. By asking the right questions at the right place at the right time, a net can thus learn to express correct features 91 29 Although the information can 'degenerate" the net can still learn thanks to the strong positional-piig injected by such periodic-encodingst (94)
single starting point,
This notion of combining information coming from many locations appears have connections Kolmogorov-Arnolu superposition theorem 22 53 The single convolutional filter in APM might be considered encoder and five layered MIP decoder. Convolution filter also be considered to be tape 92 on which symbols are written, processed by a learning machine(MIIP)) () which speaks the correct answers. We have observed that MLP's have this ability to cluster elements in any image. This seems exciting for dense visual tasks with potential for new insights. Finally, we are led believe that the networks could be made even smaller with higher bandwidth (29 28 Of course,
it shall mean defining metric called bandwidth, where the performance of the learning machine shall be measured by a three-tuple of <parameters, accuracy, bandwidth> [28 2].
Knowledge- knowledge-transser can then be a consequence of sharing folded-entn embeddings which could grow to form dynamically connected-networks rather than mimicking unfolded-outputs among multiple neural-nets. Knowledge transfer between nets of same structure is as simple as copying weights from another thereby making them immortal. There already exist approaches which share trees share knowledge between different neural nets, for e.g. dropout. A higher bandwidth way might involve exchanging folded network-vectors in higher dimensional space, which then reduces the setting of distributed federated-learning that could run in low-cost hardward"
A Turing machine gives sense of closure 68|: : the input tape is shared for both input and output to the machine. However, existing neural nets are mostly bottom-up. with feature expression limited to last layer of the net. In contrast, neurons of boltzmann machine were clamped, to allow data- vectors to be presented to the net via the environment as well as express the generated perceptual Modelling this closure
codes [1] sampled from the net itself presents problems for training a neural net: for backpropagation cant work in circles of synapstic connections, even though there mounting evidence of such connections in the brain 28). Modelling top-down influences in GLOM then has to resort to leveraging top-down influence in previous time-step influence lower embeddings estimated in the current time step via auto-encoder or neural field. A key challenge then remains to propogate top-down influences in the current step, for eg. via a recirculation proocuuuu 38 which could be trained via backprop or some other learning algorithm we are yet to discover 28 and at the same time entirely avoid the representationall/mdde
same of neurons 35 37
collapse which comes from such local forms of learning 20 29 64
B Future Work
APM offers a fresh perspective towards by--ey asynchronously 29). shall be very-exciting to see APM's potential on dense-tasks,
i.e. patches can now be procese
It can be decentralized.
17

---

video-und understanding 63 and alternative resting-schene i.c. few-shot scenarios or testing with
Another direction might involve making test-time- training faster/more- by exploring alternate zeroth-order optimization techniques [59] 288 Finally, APM contains a local-field field which emerges as a consequence of :iddng-unfolding:: this might have potential-applicatin to generative 20 45
10 /drrmmmmigg /sleepissseeeeeeeeeeeeae machine g)
C Pseudo-code for APM's operation.
In Algorithn we have inflated the entire pseudo-code to train APM beyond the applications of test-time- cest-tim-trraiing. The idea is that given input image тą APM can learn predict its entire feature grid fk and its rgb logits RGBB. First, the net inputs an image 2k-- 2y is then routed to a trigger column T. T then brings several columns Ti into existence. Each of these columns is fired into the MLP to yield location- -wwrre features fij. The loss is collected for all locations and backpropagation then estimates the gradients required to update the parameters of the APM. In this entire process, there were no labels being used. The feature grids could have been any layer of a net like DINOv2. APM thus manages learn perception field within itself 229 It can be then frozen, and used as computational equivalent of any feature-extractor for downstream-perceepeee task.
Algorithm I: Training APM in self-supervised manner using teacher U.
Data: Input data X, Student SAPM, 0APM, Teacher U (frozen), Learning Rate for each epoch do for each data point æx in X do fu + (xxx); ;
3ik (create trigger col col
IK E E.R R Route sample a column
Ti 5
Unfold
3.4 1<is
w: /+ Create location-specifal queries 29
for in range(H) do
for j in range(W) W) do
f. - (Tp):
/+ Feed-Forward column Tú into APM and decode
f6 229
+/
RGB,,
RGBBuead (Ty) 5.1 /* Decode lover-level RGB, 29
T, = L2(fí fwii);
LAGA = LL[LRGBij Ikty];
L= LAGBi
Compute Vorru
0APM < 0APM - 1) NAPMLLL
end
T fold
T 3.4
/* Collapse all location-specifay queries Ty into 2 +/
end
end
Furthermore, present the pseudo-code of APM for test-time-training in Algorithm First, the textual encoder of the teacher is used for estimating representations of each ground truth class. Then over multiple ttt-iterations, the predicted feature of APM, a.ka. is refined via statistical running-average") During each learning iteration, the trigger column is undergoing phases of folding-unfolding Finally is being used to perform zcro-shot-clasiifiation with prior-computed representations Ryt.
question may be posed how decide the optimal number of ttt iterations to achieve optimal performance. Indeed, one might build additional inductive-bias a student (aka APM) to estimate when its own Pontasie/4T//predddee ically/recurse until kickoff. Notions on soft decision-making for higher-level cognition are subtly embedded in 3
semantic-featur better than the teacher's and stop dynam-
'They are not always sleeping, they sometimes wake up too. "Averaging is similar to pooling in convolutional nets and might lose important information over time. It might be helpful explore temporally- weighed averages alternate type of long term memory- banks [100).
18

---

Algorithm 2: Pscudo-Code for operation of APM during Test-Time- Test-iiim-Trainnng..
Data: Input data X, Student SAPM, fAPM, Teacher U (frozen), Learning Rate Compute text representation of gt classes via 29
Ryt (classaame)
for each test sample In X do
J& < U(xx):
OAPM (), /* Draw net's weights for appropriate initialization. 17
21
*/
< None:
for each iteration t do
T CNN
(create trigger column 3.2 E R
/+ Route sample in column
T 29
*/
Ti Unfold
T 334 IiiS w: / Create location-specific queries 29
L. 0;
for in range(H) do for j in range(W) W)
fg 5APM((is))
Feed-Forward column Ti into
29
statisticalkummnnneenee (fa) Estimate Loss L+ = Le fkiz);
end end
Compute VAAPM L; 8APM 8APM "VAAPM
3.4
/a Collapse all location-specific queries T; into T [2 */
return pred <- contrastiveaawiieeiidd (f', Rgs):
end
D Reproducibility Statement
In order ensure the reproducibility of our experiments, we have shared the model in supplementary during review process. The code, model weights shall be released post-review. APM can work with single GPU like pascal less than GB of memory. It can also parallelize cluster containing nodes of 8 A6000 amperes. We have provided details of hyper-parameters used in test-time-training (Tat7). and precise details each layer of APM G).
E Implementation Details
E.1 Architecture and Hyperparameters
Architecture: We inflate APM's architecture in Tabld For the TTT experiments, APM consists of only single convolutional layer, and 5 MLP layers. Additionally, APM consists of feature projection head containing single linear layer, and an optional RGB head. It maybe noted that the number of kernels in the convolutional filter only This creates subtle issue: the RGB reconstruction in Fig4 is black and white. This is because single kernel loses RGB channel information. Put simply, assume a tuple
numbers representing RGB values, 1,2,3 > < 1,1,1 >. For convolution operation with single kernel assuming unit weights, the answer is 6. If this 6 gets injected in the net, is equally certain that the input was < 1, 1,2,3 > or < 4, making reconstruction from the RGB head of APM difficult. We found that this symmetry-breakee issue could noticeably fixed with Aeernela 2 Rehannelas where ncchenness the number channels in the input. Historically, various other htt.:o////rranslaitoaamammoooooo important role in designing boltzmann machines 82) The architecture in Table 6 is then meant to showcase APM's potential to extrema: how much can it do with single convolutional filter?
have played an
utilized for APM during test-time-training are detailed 7 We leveraged the seed 42 in most of our experiments, and also conducted experiments with multiple seeds. The weight matrices APM were initilized with from a random distribution with μ = 0 and 0.01. All the code has been written in Pytorch version 1.13.0. We also note that performing
Hyperparameters:
19

---

Table 6: APM architecture for TTT: with imput dimensions h, w, c and feature dimension dp: dimensionality of positional encoding. s: stride of convolutional filter in encoder, de dimension of the CLS token of teacher on which APM learns.
Layer
Feature Dimension W x C)
Aikerneta
Stride
Padding
Input Output
Input
h w x
Encoder
Conv
h/s w/s x
0/0
Linear
(dp d) * 4096
Linear
4096 4096
Linear
4096 4096
Linear
4096 2048
Decoder
Linear
2048 1024
Feature Projection Head
Linear
1024 de
Linear
(dz | d + 1024) 4096
Linear
1024 + 3 256
Linear
256 256
RGB- Head(optional)
Linear 256 3
test-time-training with 16 bit floating point allows to effectively use recent GPU architectures for eg. Ampere: they contain a larger number of tensor cores in addition to CUDA cores which results in significant speedups during the exprimentation process. Finally, normalize an input image using standard Imagenet stats, and dont resort to any other form of augmentation, thereby making the pipeline far-simpler.
Table 7: APM hyperparameters during ees-timme-traningg
Number of Test samples
50000 (Imagenet Splits), variable for other datasets.
Testing iterations Batch Size Learning Rate Optimizer
20
1
le-4
Adam
Feature Output size d Positional Encoding size
768/1024
768/1024
Image/Crop Size
448
Augmentations
Normalization, μ = (0.485, 0.456, 0.406), a = (0.229, 0.224, 0.225)
Precision
fp16 (grad-scaled)
Num of Workers Operating System
Ix rtx a6000 48GB/96GB ram/Ubuntu 22.04/2TB ssd/5TB HDD
VIT Encoder:
During our experiments in test-time- test-time-training, APM relies on
CLS token distilled from teacher trained on largessaleedattser often via contrastive image- objectives. We showed quantitative results with CLIP, OpenCLIP and qualitative semantie-clusterings with DinoV2.
CLIP zero-shot model from OpenAI which contains a vision encoder, and a textual encoder. The textual encoder tokenises input class names to features. Both image/text encoder project them to common dimensionality, and classification happens by measuring distances in contrastive space, thereby offering higher degree of freedom, opposed to training class- linear-probe. CLIP VIT-L features an output CLS token of 768 dimensions, while CLIP VIT-H outputs 1024 dimensions both of which have been accommodated in Tat (2 is foundational-modae trained via SSL-objectives and predicts significant semantically representations, which are widely used in various downstream computer-vision tasks.
Inductively, both CLIP/Dinov2 rely on VIT, which operates on the principle of parallel attention [94] image-tokens a.k.a patches can flow along parallel paths among different layers stacked over each
20

---

of APM other without any loss of spatial-resolutin which was also a key-shortcoming of convolutional Even though the transformers have no bottleneck issue, the attention -pperation in each layer still occupies significant amount of memory 64
E.2 Datasets:
To evaluate model's robustness to distribution shifts, it is necessary test them datasets which contain corruptions on a variety of scenarios for eg. rain, snow; etc.. One standard practice has been take the test set of larger datasets like ImageNet, and create synthetic corruptions establish herrhhae--encharksss on which the performance of these models can be compared. Alternatively, certain test splits have been manually-curatee from-scratch over the internet, for eg, sketches, artistic- drawings etc. Below, we detail some of the splits which were used in this paper for APM's robustness experiments.
Cifar-10C: a test-split consisting of 10000 test-samples of Cifar-10, corrupted with 15 noises, across 5 levels of poise-severities In this paper, have shown results the highest level, a.k.a level 5 owing to the resource-comee constraitss
ImageNet-C is a dataset split for recognizing objects under distribution shifts, with 1000 classes like original Imagenet. This split contains 15 types of corruptions, with each type containing 5 level of noise severities, aka, the percentage of the image region which is being typically impacted by the corruption.
Imagenet-C:
ImageNet-Ve V2: is an independent test set containing images sampled from naturally occuring scenar- ios, with 10000 images of 1000 ImageNet classes. ImageNet- -2 typically consists of splits, with varying levels of difficulties.
ImageNet-A: refers curated test-set containing "natural occurring adversarial samples", which were misclassified by the Resnet-50. This particular split contains 7500 images of 200 ImageNet categories.
ImageNet-Ro refers to a novel test-set of several Imagenet categories, which contain artistic renditions. There are 30000 images in this split spanning across 200 ImageNet categories.
ImageNet-Sketch: is challenging test-split which consists of only black-and-white sketches of 1000 ImageNet categories. This split originally consists of 50,000 images in total.
Typically, methods like CLIP evaluate these ImageNet using a prompt-ensemble of 80 handcrafted templates. APM results were also shown using this ensemble for fair comparisons.
F Additional Ablations
Here we perform some additional ablations to understand how varying one of the parameters of APM, while holding others syper-parametedd constant impacts the performance during test-time-training.
Effect of varying number of ttt iterations: We perform varying number of ttt iterations, evaluate the performance of APM on three seeds, and report the mean and standard deviations. We note that at t 25, APM obtains 49.5% accuracy with minimum observed coindard-deviatinn of 0.2.
Table 8: Ablation with variable ttt iterations on DTD dataset: We pick the best performing 53M param net in tab 7 of the main paper. We show mean/std over runs with seeds 0/7/42. The net settles on the best result of 49.5 and std reduces to 0.2 at Water 25.
Pleer APM (53M nct)
15
20
25
44.2/0.6
47.7/0.7
49.1/0.5
49.5/0.2
48.3/0.6
46.3/0.1
Effect of varying number of parameters: In Tat9 we perform an additional ablation: the number of parameters inside APM's linear layers are gradually changed from 7M to 120M. We perform TTT iterations on the DTD dataset. As can be observed, the top-
increases from 47.0 0 to 49.1, 1, thereby indicating that a 53M net was optimal for this particular instance of the problem. Beyond that, we observe gradual drop the performance, thereby indicating that the net has started to overfit. In an ideal scenario, we would want that lower number of parameters should yield higher performance. However, this would then require some more fundamental changes which allows the net to achieve higher- bandwidss, which remains matter for the future work (28).
21

---

Table 9: Ablation on APM parameter count on DTD dataset: Increasing the number of parameters to 53M improves APM's performance to 49.1 beyond which it starts to drop. Top classification accuracy is being reported.
7M
25M
36M
53M
70M
87M
104M
120M
APM 47.0
47.5
48.1
49.1
48.4
48.1
47.8
47.0
Effect of removing the teacher from APM: APM can perform test-time-training on single test- sample by relying on a CLS-token distilled from a teacher trained on-scale. This might lend itself to the assumption that APM really requires teacher in order to learn semantically-wwan representations. In this ablation, we remove the teacher entirely and have APM perform RGB-reconstruction on COCO-train set. The L2 RGB-reconstructtoo loss on COCO-val val loss then falls to 0.0027. Training took far longer than if last-layer feature-vectors were also distilled from teacher into APM.
Higher-dimensional vector-spaces carry more bits [3), but incur significant randomness 4 Island/vector- distillaoonii speeds-up the learning-procee which otherwise might only be informed via CGB-reconstruction and take lot of [2m [9]. This helps guide APM's ship correct points in the subspace. Progressing from VIT b->h in Tat shows APM becomes more competitive thereby validating this intuition.
G Additional Results
Results on Cifar-10C: In Tat TO, we show additional results on Cifar 10-C dataset widely used in test-time-training [6. Cifar- 10C consists of 15 types of noise corruptions for all the 10,000 samples present in the Cifar- 10 dataset. As evident from the table, CLIP is not naturally robust on Cifar- 10 and achieves an error rate as high as 24.5%. This is worse than existing utt-methods like TTT-Online and UDA-SS. Therefore, CLIP VIT-L is not naturally-robust on Cifar-10c.
Using it as teacher, we get the lowest average error rate of 14.8%, thereby even improving upon the performance of CLIP VIT-L/14, Note that our APM is reinitialized with random weights after every- test sample in contrast to methods like TTT-Online which retain the weights after every ttt-iteration. Inspite of that, we get a lower eтror rate i.c. 14.8% than TTT-Online 19.1%. Another benefit which APM gains is that can directly use the textual-encoder of the Clip VIT-L/14 teacher classify on cifar-10 10 test set: this allows us to bypass the requirement of training separate dataset-specific linear probe on top of our net.
Table 10: Cifar 10-C results at highest severity level of 5. We report Error Rate. Lower numbers better. t- model acts teacher for our APM. TTT was done on test set with randomly initialized weights. APM weights were reinitialized after each TTT iteration to prevent information leakage.
Lower is better.
Method
ong
g0uss
shot
impul
defoc
glass
moto
200m
S0OW
frost
fog
brit
contr
clas
pisel
ipeg
Avg
TTL-Online
8.2
253
22.6
30.6
14.6
34,
18.3
17.1
2010
18.0
13.6
21.6
21.2
19.1
21.4
Zenoshot
Clip VIT-L/14
4.63
35.4
32.3
21.9
19.3
49.7
19.3
17.3
17.0
15.1
21.6
84
15.9
34.6
25.0
27.4
24.5
VIT-I/14
APM (Ours)
21.9
30.1
137
34.1
11.9
9.0
13.5
23.0
17.0
14.8
we perform test-time-rraining on APM for different imagenet splits across increasing levels of noise-severities. We observe that APM continues to obtain competitive performance over it's teacher.
Results on ImageNet-C: In T
14
H
Some helpful analogies
APM proposes two technical ideas. 1) The first idea is the proposed column representation 7' 2) The second idea is the aading-unfolding mechanism. However, there are several deeper non-techhical/non- scientific inspirations which motivated the design of APM. We discuss some of those, to help facilitate a dleeper-connectoon and ground our intuitions.
A biological analogy 29]: Consider how organism starts its existence from a cell. The cell is copied across different body locations. Each location possesses identical DNA. However, depending on the location, the DNA decides whether to form an eye or nose. We term this process as unfolding,
22

---

Table 11: APM's performance on ImageNet-C, level The first two rows are same as the supplementary materials of [15]. A / in P means that method leveraged pre-trained weights on clean variant of train set aka, Image-net and downstream-It on corrupted version. OpenCLIP VIT-L/14 is in general more robust. APM can surpass OpenCIIP VIT-I/14.
brigh
cont
defoc
clast
log
frost
gauss
glass
impul
JP8g
monn
pisel
sbot
X30W
200m
Average
Baseline
78.5
74.5
68.1
73.9
70.5
70.6
74.8
68.6
72.3
73.0
75.2
75.9
73.6
69.3
61.7
71.4
TTTMAE
78.9
74.7
72.5
74.7
72.9
72.2
76.8
72.2
75.5
74.5
75.8
77.0
75.9
71.9
69.3
73.1
OpenCLIP LP VIE-J114
77.3
75.4
73.5
73.1
73.5
71.4
71.9
70.2
75.1
75.7
742
71.9
71.2
66.2
APM (Oans)
81.6
80.3
78.0
78.6
76.6
77.2
75.7
75.1
79.6
78.7
76.9
76.4
Table 12: APM's performance on ImageNet-C, level The first two rows are same as the supplementary materials of [IS]. A / in P means that method leveraged pre-trained weights on clean variant of train set aka, Image-net and downstream-Irs on corrupted version. OpenCLIP VIT-L/14 is in general more robust. APM can surpass OpenCIIP VIT-I/14.
brigh
cont
defoc
clost
Fog
Frost
gaiss
glass
impul
joeg
moin
pixel
shot
snow
200m
Averuge
Baseline
77.4
71.2
62.3
51.0
66.3
58
68.6
59.2
64.9
70.4
70.6
74.7
66.2
54.2
55.2
64.1
TTE-AA
77.8
71.5
69.4
49.7
69.8
62.7
72.5
66.4
70.0
72.7
72.3
76.2
70.6
58.7
63.6
68.3
Open UP VIT-J114
766
74.
53.8
22.0
62.6
67.6
64.0
73.8
69.0
66.4
61.8
66.1
APM (Ouns)
79.4
77.3
73.1
70.0
78.6
74.5
67.8
Table 13: APM's performance on ImageNet-C, level The first two rows are same as the supplementary materials of [I5]. A / in P means that method leveraged pre-trained weights on clean variant of train set aka, Image-net and downstream- on corrupted version. OpenCLIP VIT-L/14 is in general more robust. APM can surpass OpenCLIP VIT-I/14.
brigh
cont
defoc
clost
log
Frost
gniss
glass
impul
jreg
moin
pixel
shot
SnOW
200m
Averuge
Baseline
75.8
62.7
49.5
67.1
59.8
47.6
57.1
35.0
57.4
68.6
60.2
70.1
54.3
54.7
48.0
57.6
TTTMAA
75.8
64.4
59.4
71.2
64.0
54.0
63.6
50.7
64.2
71.3
64.2
73.1
61.8
58.0
57.4
64.4
Open VIT-I/14
75.8
71.8
65.5
67.7
69.0
54.7
58.9
42.4
72.8
59.9
63.5
APM (Oun))
77.2
73.3
74.8
77,8
Table 14: APM's performance on ImageNet-C, level The first two rows are same as the supplementary materials of (15]. A / in P means that method leveraged pre-trained weights on clean variant of train set aka, Image-net and downstream- ttt on corrupted version. OpenCLIP VIT-L/14 is in general more robust. APM can surpass OpenCLIP VIT-L/14.
brigh
cont
defoc
clost
Fog
Frost
gniss
glass
impul
jpeg
moin
pixel
shot
snow
200m
Averge
Baseline
73.1
33.1
35.8
56.9
54.2
45.2
39.6
26.0
38.2
62.0
43.2
60.3
32.2
44.2
40.7
47.4
TTT-MAR
72.7
39.6
45.7
64.9
58.3
52.6
48.5
42.8
47.6
67.0
50.5
66.6
42.4
45.7
51.5
53.2
OpenCLIP VIT-L/I4
74.2
64.2
58.7
57.8
66.3
52.8
45.3
34.6
68.9
46.5
63.9
41.1
56.2
45.6
54.8
APM (Ouns)
79.2
70.1
63.7
72.3
58.6
40.4
74.1
53.0
62.5
i.e a cell 'expands' to yield an organism. Next, there is evidence of jellyfish like Turritopsis dohrnii reverting from their fully grown form to younger polyp states [71). We term this process as folding. ie. cells of an organism collapse back to the single cell it began from.
A computational analogy : We now start treating an image / as a digital organism. It starts from some compressed representation T. T' unfolds to yield the image 1. I then folds back to yield the compressed representation T. Learning proceeds by oscillating between these unfolded and folded phases. At every step, the net is trying to reconstruct image I from T. T then expected to be a dense vector-space.
A cellular-automaton analogy 99]: On surface it seems a pretty trivial matter to discuss: a point can expand and yield beautiful patterns which can either be an entire universe in accordance with the theory of big-bang, or can be reproduction of an organism from a singular zygote. But, it is funny: if you start from a point, and unfold it, then all you can get is a sphere. This appears to be true for the behaviour of light, in accordance to Huygens principle [o] However, we observe non-spherical objects around us all the time. Turing posited that the symmetry breaking in the sphere must happen somewhere while the organism unfolds: such patterns could then be explained variety of diffusion based equations 43
W'ith point on the wavefront being its own source. But the envelope still growing sphere.
23

---

Figure 7: Cifar 10 islands: Individual part-wholes are clearly observed in APM features. These features are used for downstream classification. We leverage the visualization mechanism by (64). These islands are not been hand-picked 29
This idea has been explored in cellular automaton: different replication rules of starting point can yield different final pattern Scientists then continue derive different rules which yield different patterns, which is akin to how we were resorting to hand-engineering features in deep-learning for long time. APM attempts to answer the question: Is it possible build learning machine which can start from a point, unfold, and then express correct features at the correct place?. We want then push the job of rule-learnigg what backpropogation does best. We have lost the "why" for the knowledge was encoded in the weights of the net, but we seem to have gained the ability of correct features presenting themselves correct locations. This procedure thereby represents a step towards solving Arnold's superposition theoram and Hilbert's 's thirteenth problem 29 However, backpropogation can only approximate solutions and not yet reach exact ones mmes_2)... and mathematical formulations are lost into the weights of the neural-net. We then begin imagine learning machines 92 90 which can solve a complex problem like wwwggrapphybbrakinga-iiiipp
in two phases 1) relax the system towards approximate solution 40 2) have the system spit-out which parts of the solution are uncertain, and brute-force towards remaining solution. Or, could make the loss of the Barning-machin
reach perfectly zero, thereby representing perfect solution Hard problems like recognizing faces are approximately solved as a consequence of single forward pass through learning machine. If the loss could be made to reach
11 IIterraction only between local neighbours, yet order appears. Local-interaction form local-routing. and somehow stable patterns appear without global constraint. Unfortunately, backpropogation still relies on a global constraint. However, that symmetry was recently broken (55). Perhaps, the answer then lies in how the starting point instructs the unfolding process[25]
If the degree of freedom of the machine 2 than the freedom needed to solve a given problem, the machine just keeps interpolating in the subspace among multiple set of weights that give the same answer This wastes computation even if the solution has been reached. Annealing, carly-stopping and quantizing weights

---

APM
Asynchronous Perception
Error Map
Input
Reconstruction
Parallel Perception
Figure 8: Qualitative Results COCO Val set: Our APM is trained COCO-train set, and islands on COCO-val set are visualized 29 We leverage the visualization mechanism by (64). Note that these islands mechanism as in [16.. These islands are not been hand-picked. [29]
consequence single feed-forward through the net and not an iterative routing
then could consider the problem to be perfectly solved. Solvability then happen in feed-forward phase, which for practical purposes appears to be polynomial [3]
Next, we redirect the reader's 's attention to neumann's 's theory of self-reproducing automaton His idea of reff-replicating colony was that there is an infinite source of resources a.ka. reservoir which is shared by the automaton which operate at different locations of colony. The colony uses up shared resources, does self-replication and in this way converts raw materials/matter into useful -ehaevoou.. The infinite reservoir of machines he talks about then reduces to the trigger column T in APM: since features are sampled from a same space, they automatically become aware
have been some ideas to reduce this degree of freedom. would need mechanism to dynamically adjust the degree of freedom during inference.
13 computing machine which is exponentially fast can pass the illusion that non-polynomial solution- space now polynomial,
time measured constant observer spacerr--paeeaeeee [2] However, feed-forward loses how the solution was reached, i.c. we can no longer predict precise algorithmic steps. Mathematical imprecision could still lead to perfect solution. An extreme loss of precision means that the net could then run in an analog hardware 28
25

---

of themselves, thereby making explicit attention unnecessary. This also then is same as how latents have been classically sampled in the generative models [00]. One might argue that multiple automaton although starting their lives at the same point will need to communicate among themselves, as they differ among their configurations at later point in their lifespan:"") [4]] Fluctuations in 7 are then akin to mutations. We compensate for this fact by weight-sharing the MLP across different locations in APM.
A cosmological analogy: In physics, one of the famous theories of the origin of universe has been starting from single point, and undergoing a continuous expansion (5)). There are alternate theories for eg, Conformal Cyclic Cosmology [69] which hypothesize the universe undergoing periodic eycles of expansion and contraction [73.. Drawing inspiration from these fundamental insights, the trigger column I" undergoes these cycles of folding and unfolding during the learning iterations.
I A new representation: Hinton's Islands of agreement 64, 29
APM inspired from GLOM's philosophy. This section explains the principles behind island of agreement in more detail to facilitate easier understanding. GLOM [29 assumes that each pixel of an input image contains a higher dimensional column vector over it. Therefore, for an image of h x w, there is a column vector at every location. Next, all these column vectors undergo some sort of message passing between themselves, for eg, via attention. At the end of these procedure, the idea is that the vectors belonging to identical objects should start pointing in the same directions in the higher dimensional embedding space. A cluster of such higher dimensional vectors is then called islands of agreement. To see such islands in practice, one can then apply a dimensionality reduction procedure like t-sne to 3 dimensions, and visualize the obtained feature map by backprojecting the obtained features to the RGB range of [0,255] Note that t-sNE preserves the higher dimensional spatial structure among the vectors since it fits gaussians. This visualization mechanism does not require any feed-forwarding or backpropogation through the net: the islands are already there, and clustering merely helps them to reveal their presence 15
We present the islands of agreement which can be observed now in APM in Fig 7910 Notice how the whrt-wholes in the images are clearly observable in different colours. Finally, we scaled up our APM and trained it on COCO train set. Fig 12 val set. can clearly be seen that the net develops a semantic clustering and shows the potential to do dense task. APM offers unique advantage over parallel perception (DINOv2): features at particular location can be queried serially. One can 'selectively' query the locations which correspond to partieular ob))ct[[102. instead of bringing the whole feature grid in existence and then choosing the relevant objects[5] 5 Such an inductive bias of choosing which of the input rays/columns in a neural field corresponds which object has already been explored by Yu et all [02]
8 13 visualizes the islands of agreement on COCO
In the presented islands of agreement, one can see that the parts and wholes of the object are all entangled into one image. In idcalistic scenario, the net should learn to map the whole <part, whole, relationship> triple([2).. GLOM looks at the features predicted by different layers of a transformer in different way: lower layers are predicting object parts, and higher layers are predict the full object. For now, APM has only modelled the last layer of the transformer. In an idealistic scenario, we want the net to traverse the whole up-down part-whole hierarchy as well as learn the pose-transformation matrix which can get us from one part to its whole
Let us next consider 2D image of man 7 gazing at the ground in front of him. At the object level does not make sense for embeddings of the nose to jump to the embeddings of the ground. However, it is okay to learn tranfrransoornatoon matrices which can get us from one point on the man's body to another. This means that this restricted movement has to be informed by a top-down feedback which we have not yet modelled. Furthermore, this pose prediction can happen by a big fat-net like APM 16
14 As a consequence of inherent markov-stochasticity Biologically, Darwin called it evolution 9. In machines, it can be inbuilt randomness in computing element (2, 91 90]
In this material nature, the boundaries around objects are not precise boxes. Rather they gradually blend in the background. This is known as sfmato effect which leonardo hath incorporated whilst painting Mona Lisa 61 Similarly. GLOM proposes to do away with boxes altogether (2 (4].
AA big-fat fat net being used to predict embedding at each location in the part-whole hierarchy is different from simply making only the embeddings compete among themselves [66. Replicating the same net across locations is costlier than replicating location-specinn input- queut-queiess to a shared net. Both achieve the same effect. Of course, our weight sharing is not biologically plausible, which also remains deficiency of GLOM 29
26

---

shared across locations and levels of the part whole hierarchy. This is different from capsules, which contain only a few convolution filters at a particular location. In computer graphics, the pose matrix is defined four by four matrix where the first three by three elements are rotation components, last row are homogenous coordinates, and first three numbers in the last column are camera translations. If the net predicts 4 x 4 matrix each of whose element can be any constrained number, we lose the ability to make interpretations about what the predicted matrix actually represents since is no longer in well defined world-coordinate frame. This problem is same as how the 'learnt' camera poses are aligned to 'real' camera poses in neural fields while doing bundle adjustment
Note that islands of agreement operate at the finest spatial granularity: there is one column vector for each pixel of the imput percept. However, the case of perceptual overcrowding [42], there might be some hidden part of an object whose inference might be made seeing the visible pixels around it. However, the island in our case does not model this, since agreement only established for one object at a particular location 64
We could use non-intuiin non-intuitive hungarian-matching matching to resolve order among predictions of multiple object in the scene(64] (44 [5] It is non-intuitive, for the objects in this material-word don't permute in reality, and dont choose the combination which shall result in the lowest loss. Rather the brain surfs reality (2)) The brain then is the neural net, and the objects are the external input from the environment (40. The objects/islands are already there, and reveal themselves when fixaled upon. If not, they then just collapse into superimposed quantum state, for tilted cube diamond are nothing but same object 70 The granularity of fixation then vary in practice,ak saccades. Quantum state can then arise in digital machine consequence of kolmogorov's theoram. Such effects can then happen at room temperature instead of super-cooled matter states [3]. The only difference is that a system of quantum machine, there can be a different mechanism for relaxation.
components requires gradient descent in digital machine. In
27

---

Figure 9: Cifar 10 islands: Notice how individual parts are clearly observed in APM features. These features are used for downstream classification. We leverage the visualization mechanism by [64]. These islands not been hand-picked 29
NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The first claim is that we improve on ttt's computational performance, second claim is that APM performs better than existing baselines, third claim is that APM's architecture a single CNN filter and MLP of layers. Fourth claim, the best of our knowledge is that we are providing the first evidence of GLOM's insight: i.e. percept is really a field. The first claim of computational complexity is analyzed Tab 4 and Fig 2 Second claim of improved performance over Itt methods is present in tables T0P hird claim of APM's 's architecture will evident the codebase shared with this manuscript. Final claim confirming GLOM's insight i.e. percept really is a field is evident in Figō A kind
28

---

Figure 10: Cifar 10 islands: Notice how individual parts clearly observed in APM features.
These features are used for downstream classification. We leverage the visualization mechanism by 64 These islands are not been hand-picked (29
reader can also interpolate any two images in the wild using the codebase we shall share with this manuscript in the review process.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
29

---

APM
RGB Reconstruction
DINOV2 Parallel Perception
Asynchronous Perception
Error Map
Input
Figure 11: Qualitative Results on COCO Val set: Our APM is trained on COCO-train set, and islands COCO-val set visualized 29) Note that these islands consequence of single feed-forward through the net and not an iterative routing mechanism as [16]. DINOv2 does parallel perception: Í.e. all tokens are kept in the memory. However, APM does asynchronous perception: it can predict the column vector at any location asynchronously. The error map shows the error between the grid predicted by Dinov2 and the grid predicted by APM. is mostly black which shows APM closely approximates Dinov2 grid as well as can be memory efficient. The islands shown in this figure are hand-picked 29
30

---

APM
RGB Reconstruction
DINOV2 Paraliel Perception
Asynchronous Perception
Error Map
Input
Figure 12: Qualitative Results on COCO Val set: Our APM is trained on COCO-train set, and islands COCO-val set visualized 29) Note that these islands consequence of single feed-forward through the net and not an iterative routing mechanism as [16] DINOv2 does parallel perception: Í.e. all tokens are kept in the memory. However, APM does asynchronous perception: it can predict the column vector at any location asynchronously. The error map shows the error between the grid predicted by Dinov2 and the grid predicted by APM. is mostly black which shows APM closely approximates Dinov2 grid as well as can be memory efficient. The islands shown in this figure are hand-picked 29
31

---

APM
RGB Reconstruction
DINOv2 Parallel Perception
Asynchronous Perception
Error Map
Input
Figure 13: Qualitative Results on COCO Val set: Our APM is trained COCO-train set, and islands on COCO-val set are visualized 29 Note that these islands are a consequence of single feed-forward through the and not an iterative routing mechanism as [6] DINOv2 does parallel perception: i.e. all tokens are kept in the memory. However, APM does asynchronous perception: can predict the column vector at any location asynchronously. The error map shows the error between the grid predicted by Dinov2 and the grid predicted by APM. It is mostly black which shows APM closely approximates Dinov2 grid as well as can be memory efficient. The islands shown this figure are not hand-picked(29) (2).
32

---

Answer: [Yes]
Justification: See limitations sections at the end of conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e-g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
architecture for trstaiimetraining Its computational efficiency and results have been empirically validated. Furthermore, codebase shall be shared for transparency. The theoretical explanation of the entire algorithm has been presented in the section 3 Full pseudo-code has been presented in Algorithm
Justification: The paper has presented a practical computational-effin
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
33

---

Answer: [Yes]
Justification: Model Codebase has been shared. We shall release the full repo post-review.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that
appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: To preserve anonymity during the review process, code shall be hosted on anonymousgithub and shared supplemental during review. post-review we shall release the code on github, and maintain it regularly.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https! for more details.
//nips //npps.cc/puiiiii wttp:cccpppccccceocce.c http:/uullcgguuuee/eeeeeee http:///eeeeeeeeeeuuaeiooooooooo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
34

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:| Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: We follow the standard practice followed in test-iimetrraining literature 84 15 [6). Furthermore, all the experiments are run with the same seed, thereby ensuring reproducibility. All the nets are initialized with same random weight matrices to help ensure experimental consistency.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed reproduce the experiments?
35

---

Answer: [Yes]
Justification: There is whole discussion computational complexity. Il experiments run on a same desktop-workssttionn containing 1x rtx a6000/96GB ram/Ubuntu 22.04/2TB ssd.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: read the code of ethics. no human subjects were used in this work. existing open-source datasets were used. work was done by small student in academia: there is no issue of license.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have added broader impact section in the supplementary.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
36

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [No]
Justification: we have used open sourced data. so there are no asset issues. our own model weights shall be released later. The presented model APM is very small model, with potential for going into a toaster for less than dollar, thereby making ai more accessible and useful in lives of every day people.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: yes, APM is inspired from GLOM whose reference we have added.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pateessipppswiiiieea. thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: we have introduced a new network called APM. that is well documented. this paper uses well-known datasets in the existing fest-time-training literature.
37

---

Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [No]
Justification: no human subjectes were used here.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing rescarch with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justífication: The worker was a graduate student who was paid 20 hours. But he worked 90 hours because he loved it ehh.
Guidelines:
Wait for it, says Barney. Now is the time for post-credits. The fun is not yet over. This message has been subtly embedded into this list, because apparently noone scrolls down till the bottom. Since you stayed with us, big kudos to you. The rise of mortal machines shall now begin lolzy. We shall see each other on the other end. Hiya hiya hiya. All credits go shinchan, pikachu and big godzillas. But who are they?. That's a teeny-tiny secret. We might whisper to you someday. Apparently walls also have ears.
recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
38

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
39


Reviews:
[{"review_id": "ve8cjQQ8Oa", "reviewer": "Reviewer_X1eH", "summary": "This paper proposes a computationally-efficient architecture for test-time-training. APM can process patches of an image in any order asymmetrically, where it learns using single representation and starts predicting semantically-aware features. The experiment results demonstrates the effectiveness and efficiency of the method.", "questions": "See weakness.", "limitations": "See weakness.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The experiments on zero-shot test-time-training have demonstrated the effectiveness of the proposed method.", "weaknesses": "1. The writing can be improved. The current organization makes me feel very confused of the proposed method. For example, in the abstract, the novelty should be emphasized. It seems that processing patches of an image one at a time in any order asymmetrically is emphasized, which I don't think it is a novel contribution.\n\n2. As this paper targets at solving problems in test-time-training, I think more related work in this domain should be introduced in the section of related work.\n\n3. I cannot understand why the analogy compared with biology and computation makes sense. The authors introduced a lot of analogy, which has no proof and no explanation. I think the technical soundness of the paper should be significantly improved.\n\n4. The training and test details in the experimental section should be demonstrated in details, as readers may be interested in the experimental setting.\n\n5. I am wondering if the proposed method can be extended to few-shot testing scenarios?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for helping us improve our work. Please find our responses below:\n\n```[1] On processing patches of an image one at a time/lack of novelty```\n\nWe apologize that the novelties of our work were not clear. A CNN filter in any layer could process any patch by directly sliding on that region and performing convolution. Therefore, it can be said to still do asymmetric processing. However, for features to flow in the next layer, the whole CNN filter operation in the current layer will need to be finished thereby raising a waiting issue.\n\nAPM improves upon this by 1) breaking symmetry by location-aware columns, and 2) layer skipping: which allows apm to learn a mapping from input pixels to the last layer feature of a model via co-distillation. We will highlight these aspects more in the abstract.\n\nFurthermore, as pointed out by reviewer KZWi, we will also focus on the ability of apm to recover scene-semantics from a global cls token and its potential for learning part-whole hierarchies via ssl-tasks.\n\n```[2] On improving related work by adding more works``` We agree that our related work could be improved with works from other relevant areas such as test-time-adaptation as mentioned by reviewer PHih. Furthermore, as pointed out by reviewer KZWi, we will add more  prompting approaches. We have also made a note to add an additional section clarifying the feature refinement/distillation.\n\n```[3] On the presented biological analogy in section 3.1``` We apologize if our biological analogy motivated by GLOM came across as non-technical. We will shift biological insights (sec 3.1) to supplementary. We have provided more clarifications on trigger columns, folding-unfolding operation in our responses to  reviewer KZWi. We have also added pseudo-code illustrating apm's operation during TTT (https://anonymous.4open.science/r/apm_rebuttal-F3D1/ttt_pseudocode.png).\n\n```[4] On providing precise-hyperparameters/train/test-details for reproduciblity ``` Taking inspiration from [38], we provide **Reproducibility Statement**: In order to ensure the reproducibility of our experiments, we have shared the code in supplementary during the review process. The code, model weights shall be released publicly post-review. We will also share a docker image.\n\nHyperparameter tables: Code has been shared in supplemental.\n\n| Hyperparameter           \t| Value                                                                              \t|\n|------------------------------|----------------------------------------------------------------------------------------|\n| **Number of Test Samples**   | 50000 (Imagenet Splits), variable for other datasets.                               \t|\n| **Testing Iterations**   \t| 20                                                                                 \t|\n| **Batch Size**           \t| 1                                                                                  \t|\n| **Learning Rate**        \t| 1e-4                                                                               \t|\n| **Optimizer**            \t| Adam                                                                               \t|\n| **Feature Output Size $d$**  | $768/1024$                                                                         \t|\n| **Positional Encoding Size** | $768/1024$                                                                         \t|\n| **Image/Crop Size**      \t| 448                                                                                \t|\n| **Augmentations**        \t| Normalization,  $\\mu = (0.485, 0.456, 0.406)$, $\\sigma= (0.229, 0.224, 0.225)$     \t|\n| **Precision**            \t| fp16 (grad-scaled)                                                                 \t|\n| **Num of Workers**       \t| 8                                                                                  \t|\n| **Operating System**     \t| 1x rtx a6000 48GB/96GB RAM/Ubuntu 22.04/2TB SSD/5TB HDD                            \t|\n\nArchitecture details:\nWith input dimensions \\(h, w, c\\) and feature dimension \\(d_p\\): dimensionality of positional encoding. \\(s\\): stride of convolutional filter in encoder, \\(d_c\\): dimension of the CLS token of the teacher on which APM learns.\n\n| Component        | Layer  | Feature Dimension   | \\(n_{\\text{kernels}}\\) | Stride | Padding     \t|\n|--------------------------|--------|--------------------------|-----------------------|--------|-----------------|\n| **Input**            \t|    \t| \\(h X w X c\\)  | -                 \t| -  \t| -           \t|\n| **Encoder**          \t| Conv   | \\(h_s X w_s X d) | 1   | \\(s\\)  | 0 / 0       \t|\n| **Decoder**          \t| Linear | \\((d_p + d) X 4096\\) | -                 \t| -  \t| -           \t|\n|                      \t| Linear | \\(4096 X 4096\\) \t| -                 \t| -  \t| -           \t|\n|                      \t| Linear | \\(4096 X 4096\\) \t| -                 \t| -  \t| -           \t|\n|                      \t| Linear | \\(4096 X 2048\\) \t| -                 \t| -  \t| -           \t|\n|                      \t| Linear | \\(2048 X 1024\\) \t| -                 \t| -  \t| -           \t|\n| **Feature Projection Head** | Linear | \\(1024 X d_c\\) \t| -                 \t| -  \t| -           \t|\n| **RGB-Head (optional)**  | Linear | \\((d_p + d + 1024) X 4096\\) | -        \t| -  \t| -           \t|\n|                      \t| Linear | \\(1024 X 3 X 256\\) | -             \t| -  \t| -           \t|\n|                      \t| Linear | \\(256 X256\\)   \t| -                 \t| -  \t| -           \t|\n|                      \t| Linear | \\(256 X 3\\)     \t| -                 \t| -  \t| -           \t|\n\n```[5] On extensions of apm to few-shot testing cases```\nWe appreciate the reviewer's forward-outlook of apm's application beyond ttt (line 216 of paper). We will add an additional discussion on few-shot-testing in future work. \n\nWe will be happy to respond to further clarifications. Thank you so much for your valuable time, \n\nYours Sincerely, \n\nAuthors\n\np.s. references follow paper order", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FqBZk9v4ZS", "reviewer": "Reviewer_doB6", "summary": "The authors propose a new test time training method (architecture + self-supervised task) called Asynchronous Perception Machines (APMs). APM is computationally efficient, and empirically matches or improves performance on OOD image classification tasks versus prior test time inference approaches. The approach considers each $c' \\times h \\times w$ feature vector in the output of an image encoder as a c' dimensional mapping of the input spatial features to this 'island of agreement'. A convolutional neural network is used to generate this from the image. Once generated, this location aware column is folded/unfolded at inference time based on the input patch, and used to reconstruct the RGB and location specific features. The features are then averaged and compared against a text feature vector to perform classification. APM processes image patches one at a time in any order, thus also exploiting the benefits of transformer style models. Empirical results show that APM does well on out-of-distribution image classification benchmarks and exhibits competitive performance compared to existing TTT.", "questions": "Typo L17: Neural can now -> Neural nets can now", "limitations": "Discussed briefly in Section 8", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The method proposed is very novel, and (in my knowledge) the first practically viable and useful approach towards exploiting object part heirarchies as they were proposed in the GLOM paper. There are a lot of moving parts, which may or may not sustain in future work, but the key ideas (collapsed/expanded feature vectors, contrastive objective for tti instead of manually defined pre-text task) all seem like steps in the right direction. \n- The authors should be commended for reproducible science, reading through the code and instructions accompanying the paper makes it easier to understand the novel model better. It also helps establish the credibility of their work, and since this is a nascent topic will engender future research.", "weaknesses": "- While I think test time training is an important and useful paradigm, the illustrative example of \"self-driving car trying to make sense of the visual world while it is raining, but there were no such training scenarios in its training data\" (L20-21) does not really apply because tts does not really allow for instantaneous decision on a new test instance.\n- I don't think the analogies in 3.1 are anything beyond that, and as such don't belong in the main paper. It would be better to draw real scientific analogies in the main paper, e.g. to 3D novel view synthesis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and efforts in evaluating our work and providing positive and constructive feedback. We will be happy to address your comments:\n\n```Illustrative example, “self-driving car trying to make sense of the visual world while it is raining, but there were no such training scenarios in its training data”```\n\nWe thank the reviewer for this insight, and we will replace this with a better example. We agree that TTT decisions are not made ```instantaneously``` in one iteration, in the cases where the net was not-pretrained on any source data, which also happens to be the setting where APM was evaluated. Some of the experiments did showcase APM being able to semantically cluster on COCO-val, after training on large datasets like COCO-train (Fig 5). It would be an interesting future-work to see optimizations such source-training brings towards reducing required ttt-iterations. APM is also computationally efficient (tab 4) compared to other baselines. It would be very interesting to evaluate its performance leveraging other zero-th order optimization techniques [R1,R2]. As [R3] also points out, such efficient-optimization would indeed be ```a good application``` for helping make instantaneous-decisions. We thank the reviewer for this observation, and will discuss this in future work also. \n\n```Analogies in sec 3.1```\nThank you for this comment. As also pointed out by multiple-reviewers, we will shift these analogies to supplementary. We thank the reviewer for the suggestion to link our insights to 3D novel view synthesis. Indeed, given a single 3D-spatial coordinate of a pin-hole camera, neural fields operate by shooting i.i.d rays into the scene and leveraging the MLP to decode rgb. In a similar way, APM leverages location independent columns, a shared MLP, to decode location-specific features. While neural fields work for a 3D scene, apm works for a 2D percept, on a collection of images. We will add such a technical-analogy to our paper.  We will correct the spelling mistake on line 17, and add the word 'nets'.\n\nWe thank the reviewer for giving us their valuable time, and will be grateful for the chance to engage in further discussions.\n\nSincerely, \n\nAuthors\n\n\n[R1] Hinton, Geoffrey. \"The forward-forward algorithm: Some preliminary investigations.\" arXiv preprint arXiv:2212.13345 (2022).\n\n[R2] Malladi, Sadhika, et al. \"Fine-tuning language models with just forward passes.\" Advances in Neural Information Processing Systems 36 (2023): 53038-53075.\n\n[R3] Sun, Yu, et al. \"Learning to (learn at test time): Rnns with expressive hidden states.\" arXiv preprint arXiv:2407.04620 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mnLUiBiUIp", "reviewer": "Reviewer_KZWi", "summary": "This paper introduces a new algorithm for test-time training where the “test-time” task is overfitting for per-image CLIP / DINO feature distillation. The associated “downstream” task involves directly using the per-image network feature to perform image classification using dot product in CLIP space. Given an image, the method first proposes encoding it into a global feature vector via convolution. The global feature is then unfolded into per-pixel feature vectors by concatenating positional codes to the global features. These local features are then decoded and aggregated before being supervised with the CLIP image features distillation loss. This process is repeated for multiple iterations, as the global feature is “refined”. The final global feature turns out to be competitive or better than the base teacher model for some tasks and outperforms baselines. Despite the global CLS-token only supervision, the local bottleneck learns useful dense semantic features which are qualitatively visualized. The paper also shows some interesting qualitative findings when the method is trained across a dataset instead of on a per-image basis.", "questions": "I have several questions. Mainly important to clarification of the method:\n\n-\tAPM compares to baselines such as Prompt ensembling in Table 1, which is a text query side improvement. Did authors similarly use the best performing “text prompts” for their method?\n\n-\tThe per-instance distillation loss proposed by APM is minimized when the predicted feature is exactly the same as teacher feature. The fact that the final feature surpasses teacher feature performance implies that the loss does not go to 0 at training. Intuitively, what is the key constraint in the network that prevents loss from going to 0? What do the training curves look like for some example runs? \n\n-\tSummarizing the questions asked in weakness section 1:\n\no\tCould you clarify what exactly does the folding process look like? \no\tIs $I_xy$ (patch level info) appended to $T$ and fed into the decoding MLP? \no\tIs the model actually auto-regressive or purely sequential / iterative?\no\tWhat is the correct pseudo-code corrected with the 20 iterations? How is T (the encoded global feature) updated after one iteration of encoding?\no\tHow does performance on classification degrade with fewer iterations?\n\nAnswers to these questions are important for me to fully understand the paper.", "limitations": "The limitations section as currently written is a future work section. As mentioned in the “questions” section above, dense tasks and utility of local features is indeed interesting as a next direction. Additional things that can be discussed in the limitations section is the need for the method to do multiple iterations of test-time training instead of simple feed-forward inference or a single gradient step like other test-time training methods, and the variability in performance as compared to baselines.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "-\tThe paper shows that with their method, it is possible to recover dense pixel level features that are semantically meaningful (eg. features that recover object-ness, boundaries, etc) using only a global CLS token as supervision. I believe this finding is very interesting and has consequences for works even beyond the applications shown in the current paper (eg. for 2D -> 3D feature distillation)  From a representation learning perspective, this finding also reinforces the insight that the global CLS token contains useful geometric information about the image it encodes and that this information can be recovered using the right inductive biases in the distillation process. \n-\tThe paper makes interesting biological analogies to GLOM to motivate it’s iterative “folding” and “unfolding” approach, and further connects it with its reasoning on using positional codes to break symmetry in the encoding process. Figure 3 shows that over time, the local features form islands of self similarity, as highlighted / predicted in GLOM’s model learning system. The result of learning local features and islands only from global supervision does indeed seem relevant to learning part-whole hierarchies in self-supervised tasks. \n-\tlanguage-based zero-shot classification is the predominant metric chosen to evaluate the method and the paper does a thorough and sound job of running a wide range of experiments. The paper also compares with all representative sets of experiments, including a) directly using the teacher model b) other test time training approaches that optimize with a self-supervised task before prediction and c) prompt tuning approaches that are few-shot trained on the dataset. There is also thorough evaluation on a wide range of distribution shifted and cross distribution datasets. The method performs competitively and strongly on many of the benchmarks, and when it is outperformed by baselines, these runs are also duly noted, discussed and acknowledged.", "weaknesses": "I believe that the paper would benefit from some re-organization, clarification and writing improvements. At different points in the paper, there seems to be contradictory information regarding the nature of the method and some key details necessary to the success of the method are left out. Bulleted below: \n\n-\tIn the method figure (Figure 1), a flattened rgb patch seems to be appended to the flattened image encoded with convolution, annotated $I_{xy}$. However, no references to this variable are made in the methods section. Further the methods section only mentions concatenating the global feature with the positional codes, without appending any patch specific information while querying the field. \n-\tIn the same figure (Figure 1), the “folded state” in the yellow box seems to be the one that is fed into the MLP for decoding. However, in the methods section, it is claimed that it is the unfolded state that is sequentially fed into the MLP for decoding. \n-\tIn the figure and on lines 65, 129, the method makes a reference to auto-regressive querying. I’m not sure what this means, as any way in which the model keeps “state” or “memory” is not clearly described. Further, it is not described how the output of any one MLP call is fed back into future steps. After looking at the appendix pseudocode, there does not seem to be any auto-regressive component. Was the intention to mention just sequential (and not auto-regressive) encoding?\n-\tDetails about the unfolding process are clearly described, however details about the folding process are scarce. In Figure 1, the folding process seems to include a gather operation, while elsewhere it is described as an averaging operation. Further, the appendix pseudocode is lacking the “multi-iteration” nature of the algorithm. \n-\tWhen additional patch level feature losses are added (line 262) in the later sections, the details for how these are supervised are not clearly mentioned in writing. \n-\tThe related work section needs to be significantly improved: it is presently under-cited and leaves out key related work in test-time training and prompting. These are mentioned and the differences described in other parts of the paper – but reorganization is required to improve readability. A further line of related work is in feature refinement / upscaling / distillation, and such papers need to be cited and compared to in order to elucidate differences and potential novelty in distillation method. \n\nThe paper contains several unsubstantiated claims that weaken the otherwise interesting results. Perhaps, they can be clarified better, or removed:\n\n-\tThe paper claims that APM confirms that “percept is a field” as speculated by GLOM. While the paper does show that field-based decoding improves image classification, and that local features are emergent from global features, I believe a more general purpose evaluation of a broader range of perception tasks, including dense ones, must be done before this insight is claimed. As such, I believe this statement needs clarification as to what subset of ideas of GLOM are actually being “shown to work” in this paper. \n-\tThe paper claims that APM doesn’t need dataset specific pre-training (Eg. line 176). While this is true in the sense that APM is a test-time instance method, APM does require access to highly meaningful visual representations (Eg. DINO / CLIP) that have been trained at scale. APM performance, as shown by the tables, degrades as the quality of the target representation degrades. I believe paraphrasing this claim is important to reflect this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for giving us their valuable time. Please find our responses:\n\n``` On using the local patch variable $I_{xy} in the fig1 , but not in the methods. ``` Fig 1 meant to be a general case of APM's operation. In methods, APM doesn't use $I_{xy}$. We were showcasing that positional codes break symmetry even without additional patch-prior and disentangle information from global $I$. Ablations in Tab 5b), validate local-patch-injection improving from 96.5-> 96.8 on C-10 .\n\n``` On Whether unfolded state is being feed-forwarded through mlp or the folded state?``` Trigger column T exists in the folded state after a fwd pass(fig 1b), when knowledge has 'collapsed' into the weights of conv layer. When fwd-pass begins, T opens up (unfolds), and 'all' columns are forwarded through  mlp. We apologize, since the middle column in fig 1a) is shown as folded and lent itself to confusion. We will update it.  \n\n```On memory/states and auto-regression in APM``` APM encodes memory in 1) weights of T, and 2) weights of MLP. **Both** T and MLP are shared across **all** the locations. Weight-sharing induces the distributed-memory to learn relevant synaptic-patterns [11].\n\nAuto-regression unrolls a shared-decoder over time. In contrast, APM holds the whole sequence in $T$, and directly hops onto a space/time-step [R1] via a location-column. Recurrence/feedback-loops are compensated for by a form of feature-expression ([12] & our changes over GLOM [10] sec 2.1, para 2. Positional code gives the notion of distance from starting point T, and iid columns help ensure parallel-processing of columns as in the third line of fig1 caption).\n\nWe convey this by choosing ```sequential```. We will replace `auto-regression` in fig1a) and line 65.\n \n```On Details of folding-process/the gather step``` Folding operates at ```input``` of APM at the ```end``` of fwd-pass. Since a column $T_{ij}= (T|p_ij)$, and $p_{ij}$ is governed by hard-coded sinusoids, each $p_{ij}$ can be thought of as being ```annihilated``` during folding[14-16], aka deleting $p_{ij}$. Gradients from all $T_{ij}$ then flow back in towards the (center-of-mass[R3]/reservoir[R4]) column T[10].\n\n'Gather' happens during the unfolded phase ```at mlp output```:  for a particular generated column $T_{ij}$, the net is predicting the feature $f_{ij}$. Gather estimates statistically running average of $f_{ij}$, as different $T_{ij}$ get  pumped. Gather's output is used for final classification.\n\n``` On Sources of patch-level features during supervision```: We will clarify that patch-level features are the last-layer of a teacher and supervised via $L_{2}$ constraint even further in ablation section(line 231).\n\n```On APM's claims```:**percept is a field**: We will rephrase as \"APM is a step towards validating if input percept is a field\". We will revise line 176 as: apm can work with randomly initialized weights, but requires a single representation distilled from a model trained at scale.  \n\n```On apm's reliance on a teacher```: We performed an additional experiment by removing the teacher from the apm, and doing rgb reconstruction on coco-train. L_2 rgb loss on coco-val fell to 0.0027. Training took far longer than if feature-vectors were also distilled from a teacher. \nHigher-dimensional vector-spaces carry more bits[R5], but incur significant randomness. Supervision from a stronger teacher compensates and guides apm to correct points in subspace. Progressing from vit b->h in tab1, makes apm more competitive.\n\nWe will add limitation: a.k.a reduced performance as cls token's dimension reduces.\n\n```On Using prompt-ensemble on the textual side```: apm's imagenet-experiments in tab 1 used 80 hand-crafted prompts, same as CLIP. Openclip vit-h baseline also used the prompt ensemble, where apm also obtains competitive performance.\n\n```On Intuition behind final apm features surpassing teacher, stopping constraints```: APM relies on distillation, where students have been observed to outperform their teachers. In fig 3,  over 250 ttt iterations, the loss dropped from 1e-3 to 1e-12. We ```had to``` reduce the learning rate by a factor of 0.1 every 50 iterations. Whilst small, indeed it is not perfectly zero. One reason might be finite 64-bit precision of neural synapses and low learning rates. We ```waited-long``` to creep gradients into the net, as in  [11]'s footnote 20. \n\nOne might build additional constraint in a student (aka APM) to estimate when its own fantasies[18]/predicted semantic-features are better than the teacher's and stop dynamically [R6]/recurse. Notions on `soft` decision-making for higher-level cognition are embedded in [R4,R7].\n\n```On Multi-iteration pseudo-code for ttt```: Please find an anonymous code-link[Pseudocode](https://anonymous.4open.science/r/apm_rebuttal-F3D1/ttt_pseudocode.png). We will update the paper.\n\n```On Quantitative effects of lower number of ttt iterations```\nWe provide the ablation in global rebuttal pdf in Table 1. \n\n```On Improving related work```\nWe will discuss more ttt/prompting/tta approaches. We shall add differences w.r.t feature refinement/distillation: a.k.a $L_2$ mimicking feature grids vs boltzmann-temperature-matching logits[3].\n\n```On Extending limitations section```We apologize if our limitations came across as optimistic future work. We have noted APM's limitation: APM requires multiple iterations for test time training, which might increase inference time in time-sensitive scenarios. It would be very interesting to evaluate its performance leveraging other zero-th order optimization techniques [11].We also report mean/std of ttt on dtd with variable its/seeds in global pdf. We also provide more results on imagenet-c  with additional noise severities (1-4).\n\n```On Adding future work``` We will add apm's potential on dense tasks.\n\nYours Sincerely,\n\nAuthors\n\np.s please find references in comments", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qAjHFlzTMK", "reviewer": "Reviewer_PHih", "summary": "The authors propose a novel test-time training method called the Asynchronous Perception Machine. This method leverages knowledge distillation from other pretrained networks, such as CLIP. The main contributions of this approach are its robust accuracy in classifying corrupted images and its computational efficiency, requiring fewer FLOPs compared to other models.", "questions": "It is very unclear what component in the Fig 1 is. Specifically, what computations do they do?", "limitations": "There is no concerned limitation.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper presents a creative and novel neural network architecture, demonstrating strong empirical results.", "weaknesses": "1. Lack of Related Works: The paper proposes a novel test-time training method but references fewer than five related works. However, there are more than 50 papers on test-time adaptation methods published in recent years (e.g., [1-8]), none of which are mentioned.\n2. Lack of Comparison to State-of-the-Art Methods: The manuscript identifies TTT-MAE as the second-best method, yet this method was introduced in 2022. Since then, many methods, such as Diffusion-TTA [4] and DeYo [8], have surpassed TTT-MAE.\n3. Insufficient Explanation of Proposed Method: The proposed method is not explained in sufficient detail. For instance, the Vision Encoder is only depicted in a figure and not discussed further in the text.\n4. Misleading Information The “Trained” column in one of the tables is misleading. TTT-MAE does not require training data during the test phase; it requires an auxiliary loss during training.\n5. Inadequate Ablation Study: The ablation study lacks relevance. While the authors propose adaptation to distribution shifts (i.e., corrupted images), the ablation is conducted only on CIFAR-10 and CIFAR-100, which are datasets of clean images.\n\nReference\n\n[1] Tent: Fully Test-time Adaptation by Entropy Minimization\n\n[2] Continual Test-Time Domain Adaptation\n\n[3] Robust Test-Time Adaptation in Dynamic Scenarios\n\n[4] Diffusion-TTA: Test-time Adaptation of Discriminative Models via Generative Feedback \n\n[5] NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation\n\n[6] Robust Test-Time Adaptation in Dynamic Scenarios\n\n[7] SoTTA: Robust Test-Time Adaptation on Noisy Data Streams\n\n[8] Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for providing us the chance to improve our work. Please find our responses:\n\n```On improving  related-work``` As reviewer KZWi points out, we will discuss more recent tta/prompting/source-free domain-adaptation approaches. We have noted to add differences w.r.t feature refinement/distillation: a.k.a $L_2$ mimicking feature grids vs boltzmann-temperature-matching logits.\n\n```On Clarifications between ttt/tta, comparisons with more state-of-the arts``` We apologize for the confusion between test-time-training (TTT)/test-time-adaptation (TTA), and would like to clarify the experimental setting of  test-time-training and its connections to test-time-adapation works, relative novelty of apm, after checking the shared works. Thank you for sharing these works and we will add these works to related work too.\n\n - Existing works in TTA train on a source dataset and then adapt them for testing. APM, which follows TTT, works without such source-training by just initializing with ```randomized weights``` for only 1 test sample. TTA adapts to the target distribution (using all test samples), whereas TTT adapts to each sample independently.\n- TTA methods use clean datasets (eg c10) during train, and adapt on corrupted versions(c10c), which might  limit its potential to deploy on some other dataset [R1,R2,R3,R5,R6,R7], because they might require training dataset-specific linear-probe.\n- Some tta methods update statistics on \"batch of test-set\". APM can work with weights randomly-initialized after each sample. \n- APM is computationally efficient, since it uses only 1 conv layer and 5 mlp layers(tab 2, fig4), while some models might use higher-parameterized network like a ViT, or a diffusion-model [R1].\n\nWe observe that [R1] also shows performance for TTT and below we compare APM with [R1] on datasets where APM performed well. We note our baseline numbers match that of [R4]. APM presents competitive performance, on bigger datasets like Imagenet. It is important to note that APM is much more efficient than [R1] (1conv/3MLP vs. ViT+Diffusion model). We will add these and other comparisons to Tab 3. Thank you so much for pointing this out. \n|                                   | Dataset  |          |             |            |         |\n| --------------------------------- | -------- | -------- | ----------- | ---------- | ------- |\n|                                   | ImageNet | Aircraft | Food101 |\n|                                   |          |          |             |            |         |\n| CLIP VIT-B/16 Baseline (from [1]) | 62.3     | 22.8     |  88.1    |\n| Diffusion-TTA [1]                     | 63.8     | 24.6     | **88.8**    |\n|                                   |          |          |                  |\n| CLIP VIT-B/16 Baseline (consistent with [9])           | 66.7     | 23.7     |  83.7    |\n| Ours                              | **68.1**     | **29.7**        | 84.2    |\n\n| Diffusion-TTA | APM |\n| --- | --- |\n| 865M  | **25M** |\nWe also note lower number parameters in APM (25M) when compared to Diffusion-TTA (865M).\n\n- In some cases, methods like [R9] do show source-free domain-adaptation, but on smaller datasets. In tab 1-2, we have shown results on larger datasets like imagenet-c without imagenet pretraining on APM, as also duly noted in line 169. \n- We will be grateful to mention that APM is a fresh-perspective on machine perception, with the ability to do location-based processing. \n\n```Details on vision-encoder``` We apologize for this confusion. It refers to the ViT/Dinov2 vision encoder. We will add the explanation in the paper. \n\n```On misleading trained column in tables, confusion on ttt-mae``` We apologize for the confusion caused by this. TTT relies on two phases 1) train model with ssl task + labels on  train set. 2) test time -> use ssl task only. For eg., ttt-mae is initialized with pretrained imagenet-1k weights (sec 4.1 of their  paper) and needs ssl objective (masking) during ttt. However, APM is initialized with random weights, and no explicit ssl-task is required. We have renamed ```trained``` column to $P$ in all tables and clarified in caption: ```A ✓ in P means that method leveraged pre-trained weights on a clean variant of train set aka, Image-net and downstream-ttt on corrupted version```. We provide more results in global rebuttal pdf with updated captions. We will update tables in the main paper also.\n\n```On Adding ttt-specific ablations``` We will be grateful to present this ablation with a variable number of ttt iterations on dtd below. Next, ttt with varying apm parameters is presented in tab 7 of paper. Further, apm's property of one-sample-overfitting during ttt (line 250), was evaluated in tab 5a).\n| n_iter | 10\t| 15   | 20   | 25   \t| 30   | 35   |\n|--------|------|------|------|----------|------|------|\n|    \t|  \t|  \t|  \t|      \t|  \t|  \t|\n| 53M parameter  net (acc)   | 44.2/0.6 | 47.7/0.7 | 49.1/0.5 | **49.5/0.2** | 48.3/0.6 | 46.3/0.1 |\n\n\nAt 25th iteration, we can obtain 49.5, and performance degrades over lesser ttt iterations as pointed out by the respected reviewer KZWi. We tried to discuss the computational efficiency of apm, and some baselines in table 4.\n\n```On Questions: on components in fig 1 and their computations```\nComponents in fig 1 refer to a new architecture called APM inspired from GLOM [10]. It contains a trigger column T to which a given image is routed. The computation it does is a folding-unfolding process to yield location-aware columns. Each column queries an MLP and decodes a location-specific feature. These features are gathered on the output via statistical-running-average to yield a representation. The final representation is used for zero-shot classification via textual-encoder of a teacher in the contrastive-space. More technicalities are discussed in our reply to the respected reviewer KZWi.\n\nThank you so much for your time and we will be happy to respond to further clarifications. \n\nSincerely, \n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We appreciate the positive feedback from the reviewers for our work. The reviewers acknowledged several aspects such as the creativity [Reviewer PHih], novelty [Reviewer doB6], effectiveness [Reviewer X1eH], first practically-viable approach towards GLOM's [10] ideas [Reviewer doB6], being reproducible [Reviewer doB6], being able to recover semantics from just one global CLS token [Reviewer KZWi], potential for applications beyond TTT for general-representational-learning [Reviewer KZWi], exhaustive evaluations/comparisons across multiple benchmarks [Reviewer KZWi] and demonstrating competitive-empirical results [Reviewer KZWi].  

Based on reviewers constructive-feedback, we make further positive-amendments to our work;

 -  We provide a global rebuttal pdf, containing ablations with variable ttt-iterations[Reviewer KZWi], 4 additional tables with varying noise-severity levels on imagenet-c across all $15 \times 4=60$ noises, with corrections in table-columns duly noted in captions[Reviewer PHih]. We will update the main paper tables ( Tab 1-3). 
-  In addition to the code shared in supplemental, we add reproducibility statement, additional table for hyperparameters,  a new table describing apm's architecture in comments. We shall also release a docker-image [Reviewer X1eH] for computing machines with differing hardware-configurations.
- We have anonymously added ttt pseudo-code as an external link [Pseudocode](https://anonymous.4open.science/r/apm_rebuttal-F3D1/ttt_pseudocode.png)[Reviewer KZWi]. 

We shall update the supplemental accordingly

**Addressing paper-writing comments**

-  We will shift APM's biological-analogies (sec3.1) in supplemental [Reviewers X1eH,doB6], and clarify potential-novelties even better in abstract [Reviewer X1eH]. 
- We will extend related-work/limitations/future-work sections to tenth-page of the manuscript, and discuss more (ttt/prompting-approaches/tta/knowledge-distillation approaches [All reviewers] in related work), (applications to few-shot-testing [Reviewer X1eH], dense-tasks [Reviewer KZWi], using apm's local-field [Reviewer KZWi] in future-work).

We thank the reviewers for giving our work their valuable time. We have also provided individual-responses and look forward to the chance to engage in further-discussions. 

Yours Sincerely, 

Authors.

Meta-review: The paper proposes a novel test-time training method, Asynchronous Perception Machines (APMs), which is computationally efficient and demonstrates robust performance on out-of-distribution image classification tasks. The method leverages patch-based processing and knowledge distillation to improve classification accuracy while requiring fewer computational resources. Strengths include its novel approach to handling test-time training and the clear empirical validation of its effectiveness. The method’s ability to process image patches in any order and its competitive performance across benchmarks are notable. Please address reviewers’ comments in the final version.
Decision: Accept (poster)