Asynchronous Perception Machine for Efficient  Test Time Training

Authors: rajat  modi
yogesh  singh
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 21:56:21

Abstract: In this work, we propose Asynchronous Perception Machine (APM), a computationally-efficient architecture for test-time-training (TTT). APM can process patches of an image one at a time in any order asymmetrically, and still encode semantic-awareness in the net. We demonstrate APM’s ability to recognize out-of-distribution images without dataset-specific pre-training, augmentation or
any-pretext task. APM offers competitive performance over existing TTT approaches. To perform TTT, APM just distills test sample’s representation once. APM possesses a unique property: it can learn using just this single representation and starts predicting semantically-aware features. APM’s ability to recover semantic information from a global CLS token validates the insight that CLS
tokens encode geometric-information of a given scene and can be recovered using appropriate inductive-biases. This offers a novel-insight with consequences for representational-learning. APM demostrates potential applications beyond test-time-training: APM can scale up to a dataset of 2D images and yield semantic-clusterings in a single forward pass. APM also provides first empirical evidence towards validating Hinton at Al’s GLOM’s insight, i.e. if input percept is a field. Therefore, APM helps our community converge towards an implementation which can do both interpolation and perception on a shared-connectionist hardware. Our
codebase has been made available at https://rajatmodi62.github.io/apm_project_page/

--------

**It now appears that some of the ideas in GLOM could be made to work.**

https://www.technologyreview.com/2021/04/16/1021871/geoffrey-hinton-glom-godfather-ai-neural-networks/

```
   .-""""""-.
 .'          '.
/   O      O   \
|      O       |
 \  '------'  /
  '.        .'
    '-....-'
A silent man in deep-contemplation.
Silent man emerges only sometimes.
And he loves all.
```

Main paper: 
 introduction: 
 In these past centuries, computing-machines have become a lot faster [98] . This made it possible to train higher-parameterized neural nets and led to interesting emergent abilities [86] . As was predicted by Turing himself, and as were his suspicions of Lady Lovelace's arguments against learning machines [96] : neural-nets can now finally learn without human-feedback [7] , paint pictures [82] and even compose a sonnet [98] . Even with such impressive-progress, a key question still remains: how can these nets recognize images whose distribution is far different from the ones which were used during training? For e.g., consider a self-driving car trying to stop when it encounters a pedestrian crossing a road. Such practical scenarios require 'instantaneous-decisions' 2 for ensuring human-safety in autonomous-systems [70] .Test-time-training (TTT) [91] is one of the promising techniques for handling such distribution shifts: a neural net adapts to a test sample 'on the fly'. Since the label of the sample is not known, the net performs some auxiliary pre-text task like data augmentation [17] , rotation [17, 92] or prompt tuning [90] on it. After several such iterations, the net recognizes the test sample. The key idea is that the net is allowed to dynamically adjust its decision boundary even after it has been trained, thereby bringing it much closer to how humans keep learning 'continuously' throughout their lifespans [92] . T i then unfolds and generates h × w location-specific queries. These queries are i.i.d and can be parallelized across cores of a gpu [53] . Each query T i is passed through a shared MLP and yields the vector f ′ xy and f ′ rgb . MLP is queried iteratively until whole grid f ′ comes into existence. Classification then involves comparing the averaged representation f with class-specific textual representations in the contrastive space . (ii) Folded State: The parameters which the net learns are parameters of T and MLP. (iii) Unfolded State: T expands to yield h × w queries 'on-the-fly'. Learning involves oscillating this net between folded and unfolded states. This net can be trained via backpropogation [84, 33] .Despite the success of existing TTT approaches, several limitations need to be addressed [90, 17, 92] : 1) The Information Bottleneck Problem [70] : Multiple TTT iterations requires feed-forward through many hidden layers multiple times, making it computationally expensive. 2) Reliance on a surrogate pre-text task: the optimal data-augmentation pipeline or the best pretext task is not known beforehand, worsening the issue even further in an online setting. 3) Furthermore, TTT leverages architectures like transformers which rely on parallel perception: this requires projecting all input patches into a shared representational space, thereby consuming significant memory.Inspired by GLOM's philosophy [34] , we hereby propose Asynchronous Perception Machine (APM), a new architecture for efficient test-time-training. 1) It handles the information-bottleneck problem by directly learning a shortcut from input image to final representation from last layer of a model [33] . 2) During TTT, we compute test-sample's representation only once. Subsequent iterations involve over-fitting on this representation only and doesn't require any data-augmentation/pretext task. 3) APM can operate on a single patch at any location asynchronously [97] and still encode semantic-awareness in the net, thereby offering a fresh perspective towards machine-perception.We make the following contributions in this work:• We propose APM, a GLOM-inspired architecture that can perform test-time-training without requiring data augmentation/auxilary-tasks. APM is a step towards validating GLOM's philosophical insight: a percept is really a field [34] . • APM is computationally efficient, i.e it almost halves the number of FLOPs over existing methods [17, 92] . APM matches/surpasses TTT performance by 0.4% -8% across 16 datasets. • APM is architecturally simple: a convolutional layer and a single MLP of 5 layers. APM can still learn using one sample and semantically clusters on a collection of images [33] . 
background: 
 We draw from insights previously philosophically mentioned in GLOM [34] . Consider how machine perception has been done classically: an input x ∈ R c×h×w is transformed by a non-linear function f to a perceptual feature grid y ∈ R c ′ ×h×w , c denotes image channels, h, w are input spatial dimensions.Islands of agreement [34] Rather than viewing this matrix y as a cuboidal grid, one can now see this as column vectors v c ′ at each location. There are h × w such columns in y. Therefore, there exists a one-to-one mapping, between each input patch i.e. (x, y) with the vector v c ′ at that location. A neural net f can then thus learn to fit f (I, x, y) → v c ′ . These vectors v c ′ have been termed islands of agreement in GLOM [34] . They were recently demonstrated on realistic datasets by [70] , showing some promise that trained models can serve as free supervision-sources for vectors v c ′ .Next, we imagine feed-forwarding one patch (x, y) into the neural net f and estimating the response v c ′ . It thus becomes possible to process these patches one-by-one rather than keeping them all together in memory [100] . We then further imagine auto-regressively querying f until the whole grid y has been brought into existence.Implicit-representations/Neural-Fields: The neural net f inputs a coordinate based query, (I, x, y) where (x, y) is the 2D location in an image I. It then gives the answer v c ′ [34] . Such implicit representations have been studied extensively in 3D novel-view synthesis [65] . However, in the above problem I is a simple 2D image. In a recent work [34] , it was hypothesized that neural-implicit representations can work on 2D images as well without having to retrain it every-time.Layer Skipping [33] : We now combine the previous two insights together. Imagine that v c ′ has been estimated from a model after travelling through several layers. We can distill these v c ′ into an implicit representation f and learn a direct mapping between input x and last layer of a model [33] .During inference, we can skip feed-forwarding through all the model layers. The only feed-forward cost we then pay is what it costs to travel through f . Provided that the number of parameters in f are lesser than parameters in the teacher model, we can expect computational speedups.We now introduce a connectionist-net f motivated by these insights. It can perform efficient testtime-training on a given 2D input.Question: Does the paper discuss the limitations of the work performed by the authors? [34] . Note that these islands are a consequence of single feed-forward through the net and not an iterative routing mechanism as in [18] . DINOv2 does parallel perception: i.e. all tokens are kept in the memory. However, APM does asynchronous perception: it can predict the column vector at any location asynchronously. The error map shows the error between the grid predicted by Dinov2 and the grid predicted by APM. It is mostly black which shows APM closely approximates Dinov2 grid as well as can be memory efficient. The islands shown in this figure are not hand-picked [34] . [34] . Note that these islands are a consequence of single feed-forward through the net and not an iterative routing mechanism as in [18] . DINOv2 does parallel perception: i.e. all tokens are kept in the memory. However, APM does asynchronous perception: it can predict the column vector at any location asynchronously. The error map shows the error between the grid predicted by Dinov2 and the grid predicted by APM. It is mostly black which shows APM closely approximates Dinov2 grid as well as can be memory efficient. The islands shown in this figure are not hand-picked [34] .Answer: [Yes] Justification: Model Codebase has been shared. We shall release the full repo post-review. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
asynchronous perception machine (apm): 
 APM processes an input image I via two novel mechanisms: 1) A column module T which is said to contain an input image I, 2) a column folding-unfolding mechanism that operates during each forward-pass. We first provide a technical analogy to better understand APM.A neural field does 3D novel view synthesis by querying an MLP with a point (x, y, z) and yielding corresponding rgb. In a similar way, APM does 2D perception by querying an MLP with an image I, and a location (x, y) to yield location-specific feature f xy . APM features a new mechanism to query the MLP, i.e a column module T . Next, we define this column representation T .We define a column T as a vector of dimensions 1 × 1 × d. Our aim is to map image I in this T , so that T can summarize its entire identity. Given an image of dimensions c × h × w, we run a 2D convolution on it. Number of filters are set as accordingly. The resultant 1 × h ′ × w ′ feature map is then flattened into a single column vector T of dimensions d = h ′ × w ′ . We shall refer to this column T as "triggering hyper-column"(seed). The only learnable parameters in this column are parameters of a convolution filter.3 [98, 97] .The trigger column T now starts undergoing cycles of folding-unfolding (Fig 1 ). During unfolding, the column T copies itself to yield h × w location-aware columns. During folding, these h × w columns collapse back into a single column T . The neural-net then oscillates between these folded-unfolded phases during learning iterations.The unfolding-process shall now be concerned with generating location-aware column T ij from T . We generate h * w 2-D non-parametric positional encoding similar to the ones being used in transformers [100] and neural fields [65] . The trigger column T ij is then given by T ij = (T |p ij ), where | denotes the stacking operator and (1, 1) ≤ (i, j) ≤ (H, W ). T can be said to encode identity of an image.The folding-process involves collapsing all columns T ij back into T from which they had begun. This is achievable since the p ij used in T ij was deterministic. An abstract-mathematical intuition on folding is that all p ij in T ij get deleted/annihilated at the end of every forward-pass, and only T is left. Positional-encodings contain periodic-sinusoids which offers a strong positional-prior in practice [100] . Gradients collected from all T ij then update the parameters of the convolutional filter in the column T . This sharing of T across different locations (i, j) now induces a new fundamental behavior [34] already hypothesized in GLOM (more details in section 5.3).This representation T ij = (T |p ij ) exhibits a strong-symmetry breaking behavior [34] 4 . For e.g., consider different locations across same image I. Although the identity T will be the same, the positional encoding p ij shall differentiate among different locations. The converse holds true: given the same position p ij in two different images, the column T shall differentiate among identities of different images.All the experiments in this paper are performed without-injecting the local patch I xy in the trigger column T ij . This showcases the strong-symmetry breaking behaviour of positional-encodings [100] : they can disentangle location-information from global I without-requiring an additional patch-prior.Each column T ij is passed through an MLP to yield location-specific features f ij and RGB values RGB ij . Number of neurons in the first layer of the MLP is same as dimensionality of column T ij .Column Independence: The MLP is shared across all the columns. One column is also independent of another as illustrated in Fig1. Therefore, the MLP can be queried sequentially. Firing a column T ij into the MLP yields a column vector v c . Once h × w columns are finished firing, we get a feature grid f of dimensions h × w × c. Note that the number of columns can be as low as 1.We detail how the t th iteration of test-time-training could be performed. First, the obtained feature grid f ϵR h×w×c from APM is averaged to yield f avg ϵR c . For the first TTT iteration, i.e. t = 1, the image I is feed-forwarded through a multi-modal teacher like CLIP to get a CLS token f cls and corresponding text representation t cls . APM then learns to estimate this same target feature f cls . We enforce this by a simple L 2 constraint as:EQUATIONwhere f avg,t is the averaged output feature from APM at a particular TTT iteration t. Note that the target f cls is only estimated in t = 1 and remains the same for t ≥ 2, i.e. subsequent feed-forward through teacher is not needed.Memory-efficient estimation of f avg,t : During a TTT iteration t, f avg is computed as a simple average of f ϵR h×w×c . This would require h × w columns to exist in the memory simultaneously. APM's design assumes column independence which allows estimating f avg as a statistical running average, i.e.EQUATIONassuming, n columns have already been fired into the APM and one additional column corresponding to position (i, j) of image I is in the process of firing. This procedure is repeated until all positions (i, j) are exhausted. We represent the sequential column-firing by a 'Gather-Grid' operator in Fig1.Predicting image class-label: After certain TTT iterations, (say t = 20), the output feature f avg,t and the textual features t cls are obtained. Image-classification then follows the standard practice of comparing the distance of f avg,t with each plausible class feature t cls in the contrastive space and choosing the closest one as the prediction [81] . 
experimenting with apm: 
 We quantitatively evaluate APM on zero-shot test-time-training on popular benchmarks containing distribution-shifts [17, 92, 90] . Next, we quantitatively explore its computational efficiency.Datasets: Cifar-10C [92] contains 5 level of corruptions on the test-set with 15 types of noises.Larger datasets with significant distribution shifts consists of ImageNet val/other curated splits. For e.g., ImageNet-V2 contains natural images consisting of 10k images across 1000 classes. ImageNet-A contains 7500 adversarial-images consisting of 200 categories. ImageNet-R consists of 30000 artistic-images across 200 ImageNet categories. ImageNet-Sketch consists of black/white sketches of 50000 images for 1000 classes. ImageNet-C consisting of 15 types of corruptions with 5 levels of severity. There are additional 9 cross-dataset generalization datasets [90] .Baselines: We compare against standard TTT-online [92] , TTT-MAE [17] , TPT [90] , CLIP VIT-B/16, Coop, CocoOP. We also benchmark CLIP VIT-L/14 and the strongest OpenCLIP VIT-H/14-quickgelu variant pre-trained on dfn5b. APM's computational effectiveness stems from this unique ability to overfit on a single test sample's embedding which was distilled only at t = 1. This merits a deeper investigation. [34] : APM is overfit on a test-sample's representation distilled from a teacher. We plot t-sne clustering of output features over 250ttt iterations. L 2 loss between predicted features and distilled sample falls from 1e-3 to 1e-12. Moving left to right shows that wholes break into smaller parts. APM can learn on a single sample: In Fig 3 , we feed-forward a single image through our APM and perform 250 TTT iterations 5 . We only overfit on one distilled test-sample embedding and plot the 2D t-sne clustering of predicted APM. It can be seen that the elements of the scene have gradually started to cluster. This suggests that APM solves an inverse problem: given a single test-sample embedding, what features in the image led to its formation? Over several TTT iterations, APM's features become representative enough to explain different parts of a scene [34] . Therefore, the same-net could be made to move up-down the part-whole hierarchy [33] , although it requires TTT-iterations for now.Next, we explore if data augmentation improves APM's performance. We perform TTT iterations on CIFAR-10's test set and find that data augmentation drops APM's performance from 98.6 to 76.7. This quantitatively demonstrates that APM works best when it does one-sample overfitting. It now aligns with the earlier qualitative experiment: over TTT-iterations, the network is learning to cluster the elements in the scene3. Data augmentation distorts the sample and makes it difficult for APM's predicted features to agree on a stable, relaxed-representation that explains the scene [47, 30] Till now, APM's operation has involved random-initialization of weights for every test-sample and performing test-time-training. There is another mode that it can be made to operate in. 
apm training (qualitative analysis): 
 APM can also scale up and do learning on a batch of samples (for e.g., COCO images [55] ) distilled from a teacher [36] . This requires introducing several new mechanisms. Note that this section is meant to qualitatively demonstrate how scaling up APM can improve the net's interpretability, and help seed future research. Quantitative experiments beyond test-time-training remain out of the scope of this paper. APM's training follows a standard setup in self-supervised-learning [7] . We have provided the full algorithm for SSL-training/test-time-training in the AppendixC. During inference, APM takes any 2D image x k and predicts its RGB reconstruction RGBk/higher dimensional features f k . The net then begins to demonstrate several interesting properties, which we will discuss next. Given a sample x k , APM can reconstruct its RGB. In Fig4, we achieve this by estimating f rgb,ij = (T ij |f ij ). This skip-connection from trigger column T ij to output feature has a subtle reason: consider a white dog and brown dog. The predicted object-level feature for both will be almost identical [34, 2] . However, T ij is different for both since it contains lower patch-level features [34] . Therefore, this helps us break symmetry. Without this skip-connection [25] , the net fails to predict RGB. The network is trained to reconstruct RGB for a batch of images,L rgb = N i=1 h * w j=1 L 2 (p ′ j , p j ), where p j /p ′ j are ground truth/predicted RGB-logits respectively.5.2 APM is asynchronous yet encodes semantic-awareness in the net.Given a sample x k , APM can directly learn to mimic the entire last layer feature-grid which a teacher model would have generated. We enforce this by aL grid = N i=1 h * w j=1 L 2 (f ′ j , f grid ). In Fig 5(ii)we estimate the error map between the features predicted by our APM and Dinov2 [73] . The error map is mostly black which shows that it closely approximates Dinov2's grid 6 . Note that APM does patch-based asynchronous processing whereas DINOv2 relies on parallel perception. Finally, Fig 5(iii) shows a simple feed-forward of a CIFAR-10 sample through APM. We can see semantically-aware features. Note that this is a single feed-forward through APM [34] . Predicting output feature grid allows the net to encode dark knowledge, i.e. the knowledge of both correct and incorrect classes [35] . This is better than just mimicking one hot vector of a target class since it manages to encode lower probabilities of wrong classes also.Input Dinov2 APM Error Map5.3 APM is a step towards validating GLOM's insight: input percept is a field [41] .Source Target 1 2 3 4 5Figure 6 : APM is a step towards validating GLOM's insight [34] : input percept is a field. An interpolation between any two images in the wild. This field arises in APM's MLP consisting of 5 layers. Trigger column T acts as a key which retrieves an image from the APM's memory. T resides in a continuous embedding space, not discrete addressing space.In Fig 6,  we show that APM can interpolate between any two images in the wild. We choose two images I 1 and I 2 . These images are then funneled through the trigger column T and yield two vectors v 1 and, v 2 respectively. Next, we generate n intermediate latents separated by an equal linear distance by v j = v 1 + v2-v1 n . Each latent then brings into existence its own set of location-aware columns and decodes an image from the MLP. Such an interpolation has been previously observed in other models [22] . APM now functions as a new form of addressing mechanism: the trigger column T acts a key. Copying T across locations yields image-specific queries [34] . Values are synapses triggered in the MLP. RGB decoding happens in the output head. Hence, such continuous keys and queries exist outside the net [34] .Classically, auto-regression has unrolled a shared-decoder over time [105] . In contrast, APM holds the whole sequence I in T , and directly hops onto a space/time-step [13] by querying the MLP with a location-column T ij . Note that T ij is generated by unfolding T . Recurrence/feedback-loops are compensated for by a form of feature-expression [34] . This is a step towards validating GLOM's insight, i.e. input-percept is a field [34] and one can now interpolate in it (gestalt psychology). Furthermore, the trigger column T resides in a continous embedding space, and not discrete hardware locations(classical AI) [34] . Therefore, APM tries to integrate insights from both fields.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: To preserve anonymity during the review process, code shall be hosted on anonymousgithub and shared in supplemental during review. post-review , we shall release the code on github, and maintain it regularly. Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
ablations on apm: 
 The experiments on TTT had relied on a curious ability of APM: it could simply overfit on a test-sample's distilled representation at t = 1. This merits further investigation:In Table 5a , we investigate whether existing networks are capable of one-sample test-time-training. We employ a randomly-initialized network to overfit on a distilled test sample's token obtained in the first TTT iteration [33] . Standard MLP achieves low accuracies of 9.0 and 3.8 on CIFAR-10 and CIFAR-100, respectively. Notably, an 11.4M parameter ResNet18 outperforms the larger ResNet34 with 21.5M parameters. A reason might be that too many parameters in ResNet34 gives it too many degrees of freedom [37] : it finds it hard to overfit on one Effect of increasing number of convolution in T: Increasing number of convolutional kernels from 1 to 3 improves from 96.08 to 97.67.These ablations reveal: 1) APM can do one-sample overfitting for a test sample, 2) It helps to have both local patch and a strong image representation in the trigger column T, and 3) Increasing the levels of part-whole supervision strengthens the net.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer:[Yes]Justification: [Yes] Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
related work: 
 Prompting Approaches: Prompting is a mechanism to adapt a foundational-model to a downstream task in a zero-shot manner [63] . However, prompting typically requires well-designed hand-crafted prompts. Prompt-tuning methods consist of learnable prompts which enable a parameter-efficient approach to fine-tune a foundational-model. CoOp [113] applied prompt-tuning to CLIP. However, CoOp [113] is sensitive to OOD-data, which CoCoOp [112] in turn compensates for by conditioning the prompts on model inputs. Similarly, TPT [90] optimizes a prompt to encourage consistent predictions across multiple augmented views of the same test sample, and uses confidence-selection to filter out noisy-predictions. Note that TPT [90] performs prompt-tuning over ViT-B/16 but requires feed-forward through all the model-layers for every iteration.Test-Time-Optimization: Introduces the notion where a model adjusts its decision boundary dynamically during testing, for eg, improving robustness to distribution shifts. Test-time training (TTT) generally adds a self-supervised multi-task branch, and performs a SSL-task like rotation, or maskedreconstruction to adapt the network to the test-sample [17] . These approaches typically initialize the net with pre-trained weights, for eg, Imagenet before undergoing ttt-iterations on a downstream corrupted-dataset. An alternate line of work, for eg, TENT [102] proposes to minimize entropy of batch-wise test-samples. However, TENT [102] requires more than one test-sample to converge towards an optimal solution, whereas APM can also operate on one test sample. Another line of work adjusts internal batch-norm-stats of a network [87] . However, this makes the network-architecture inflexible and requires more than one test sample for optimization. Several other papers following the original pioneering-TTT paper [92] have worked on different problem formulations, for eg, assuming access to an entire dataset (e.g. [56, 78, 103, 110, 20, 111, 21] ) or a batch (e.g. TENT [102] ) of test inputs.In contrast, APM evaluates on a each test-sample independently. Inductively, APM does not require any dataset specific pre-training, a pretext task or prompt tuning. Some approaches also use higher-parameterized transformer/diffusion models [78] making the optimization compute-intensive. However in APM, for TTT iteration t > APM also inherits zero-shot behaviour from CLIP, which allows it to bypass training a separate dataset-specific linear-probe for downstream TTT. Finally, some works in areas like source-free domain-adaptation do perform TTT on smaller datasets like Cifar-10 etc [102] . APM additionally shows results on Imagenet splits (Tab1,2) and various cross-generalization datasets(Tab3).Part-whole hierarchies: The idea of encoding part-whole dynamic-parse-trees as distributed representations in neural nets can be traced back to [38] , with recent attempts leading to capsule networks [47, 85] . However, the EM [47] /attention-based [85] routing forces each capsule to represent only one part [70, 68, 34] . This fundamental flaw prevents capsules from scaling-up and generalizing to multiple OOD objects [70] . Recently, GLOM [34] proposed a theoretical system of representing each input pixel as containing a column-vector. These vectors then undergo a routing procedure such that pixels corresponding to same part come to 'agree' with each other. [18] demonstrated GLOM on Cifar splits. APM does not need any routing because it processes location-aware columns independently. APM now reveals an additional perceptual-interpolation property not shown earlier(Fig6).Knowledge Distillation: There has been a long history of training data-specific mixture/product of experts and having their ensemble vote towards a prediction, with the key idea to make the experts as different from as each other as possible, and only one expert deciding on the predictions of a particular input sample [51] . This is better than having all experts give equal opinion on a sample [39] .Other methods attempted to 'gather' their collective knowledge to a single model for edge-device deployment [36] . Recently, knowledge has been transferred from a larger teacher model to smaller ones [6] , for eg, a foundational model [94, 80] . The student often retains/becomes-better than its teacher [1] . In practice, the weights of teachers are fluctuated slowly (EMA) as compared to the student [93] . This mechanism then helps realize Kahnman's theory of slow-fast thinking [52] .Typically, distillation requires boltzmann-matching [5] predicted distribution between students and teachers, with the distribution's sharpness being governed by a temperature parameter [45] . In contrast, APM directly mimics the entire last layer feature grid of a teacher via L 2 norm [99] . Furthermore, APM possesses a novel-inductive bias that can recover semantic-features from a single CLS-token distilled from a teacher [Fig 5] . This validates the intuition that CLS tokens encode useful geometricinformation of a scene after cross-attention of CLS token with patch tokens of an input image [12] .Routing Mechanisms: Routing mechanisms involve routing correct object-specific information to correct neurons. [59] proposed slots which perform binding by iterative rounds of self-attention. Löwe et Al [61] proposed 2D complex autoencoder and rotating features [62] where presence of an object is encoded in phase of a neuron and follows the minimum description length principle [44] . In APM, binding is done via the location itself [34] . GroupVIT [108] routes information to multiple group tokens and shows that semantic segmentation emerges with just image-text contrastive supervision.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [NA] Justification: We follow the standard practice followed in test-time-training literature [90, 17, 92] . Furthermore, all the experiments are run with the same seed, thereby ensuring reproducibility. All the nets are initialized with same random weight matrices to help ensure experimental consistency.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
conclusion: 
 We propose APM, inspired from the insights presented in GLOM [34, 43] . APM promises to be an efficient architecture for test-time-training and asynchronous patch-processing [97] . APM shows robustness to extreme distribution shifts [28] . APM demonstrates that MLP's can be made to semantically cluster a given image. We hope that APM will help inspire further research on simpler weight-sharing, lower-memory, higher-bandwidth baby-friendly7 efficient-nets [33] . We love GLOM shoo much and everyone who keeps helping it. [34] .little_dogzilla Limitations: In this work, we have mainly-focused on image-classification. Furthermore, APM requires multiple TTT iterations for now, although they might be reduced by exploring pre-training on a source-dataset [33] . APM can work on just 1 sample with randomly-initialized weights. However, it still requires a single CLS token which has been distilled from a teacher pre-trained on a large-scale dataset. Our preliminary-experiments have revealed that APM can still do RGB-reconstruction without a teacher [26] , showing potential that APM can be self-sufficient and independent.Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [No] Justification: no human subjectes were used here.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [No] Justification: The worker was a graduate student who was paid 20 hours. But he worked 90 hours because he loved it ehh.Guidelines:• Wait for it, says Barney. Now is the time for post-credits. The fun is not yet over. This message has been subtly embedded into this list, because apparently noone scrolls down till the bottom. Since you stayed with us, big kudos to you. The rise of mortal machines shall now begin lolzy. We shall see each other on the other end. Hiya hiya hiya. All credits go to shinchan, pikachu and big godzillas. But who are they?. That's a teeny-tiny secret. We might whisper it to you someday. Apparently walls also have ears. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
 
 Appendix: There are two main ideas that led us to Asynchronous Perception Machine [45, 34] . The first idea is that instead of thinking of features as a cuboidal feature grid [32] , one can think of it as a column vector at each location [34, 70] . This helped us learn one to one mapping between the input rgb patch and the vector at a particular location. It led to the net being able to process one patch at a time.The second idea is this notion of collapsing information into a single starting point. Our previous understanding was that this 'collapse' leads to degeneracy [45] . In this paper, the information can be recovered from the starting point by copying it many times and breaking symmetry with positional encoding. By asking the right questions at the right place at the right time, a net can thus learn to express correct features [97, 34] . Although the information can 'degenerate' to a single starting point, the net can still learn thanks to the strong positional-prior injected by such periodic-encodings [100] .This notion of combining information coming from many locations appears to have connections to Kolmogorov-Arnold superposition theorem [34, 58] . The single convolutional filter in APM might be considered an encoder and five layered MLP as a decoder. Convolution filter can also be considered to be a tape [98] on which symbols are written, processed by a learning machine(MLP) [98] which speaks the correct answers. We have observed that MLP's have this ability to cluster elements in any image. This seems exciting for dense visual tasks with potential for new insights. Finally, we are led to believe that the networks could be made even smaller with higher bandwidth [34, 33] . Of course, it shall mean defining a metric called bandwidth, where the performance of the learning machine shall be measured by a three-tuple of <parameters, accuracy, bandwidth> [33, 2] .Knowledge-transfer can then be a consequence of sharing folded-embeddings which could grow to form dynamically connected-networks rather than mimicking unfolded-outputs among multiple neural-nets. Knowledge transfer between nets of same structure is as simple as copying weights from one to another thereby making them immortal. There already exist approaches which share trees or share knowledge between different neural nets, for e.g. dropout. A higher bandwidth way might involve exchanging folded network-vectors in higher dimensional space, which then reduces to the setting of distributed federated-learning that could run in low-cost hardware 8 .A Turing machine gives us a sense of closure [74] : the input tape is shared for both input and output to the machine. However, existing neural nets are mostly bottom-up, with feature expression limited to last layer of the net. In contrast, neurons of a boltzmann machine were clamped, to allow datavectors to be presented to the net via the environment as well as express the generated perceptual codes [16] sampled from the net itself on a same set of neurons [40, 42] . Modelling this closure presents problems for training a neural net: for backpropagation cant work in circles of synapstic connections, even though there is mounting evidence of such connections in the brain [33] . Modelling top-down influences in GLOM then has to resort to leveraging top-down influence in a previous time-step to influence lower embeddings estimated in the current time step via an auto-encoder or a neural field. A key challenge then remains to propogate top-down influences in the current step, for eg, via a recirculation-procedure [43] which could be trained via backprop or some other learning algorithm we are yet to discover [33] and at the same time entirely avoid the representational/mode collapse which comes from such local forms of learning [22, 34, 70] .APM offers a fresh perspective towards machine-perception: i.e. patches can now be lazily-processed one-by-one asynchronously [34] . It shall be very-exciting to see APM's potential on dense-tasks, video-understanding [69] and alternative testing-schemes i.e. few-shot scenarios or testing with a 'batch-of-samples'. Another direction might involve making test-time-training faster/more-efficient by exploring alternate zeroth-order optimization techniques [64, 33] . Finally, APM contains a local-field which emerges as a consequence of folding-unfolding: this might have potential-applications to generative [22, 48, 11] /dreaming [71] /sleeping-machines9 [9] .C Pseudo-code for APM's operation.In Algorithm1, we have inflated the entire pseudo-code to train APM beyond the applications of test-time-training. The idea is that given an input image x k APM can learn to predict its entire feature grid f k and its rgb logits RGB k . First, the net inputs an image x k . x k is then routed to a trigger column T . T then brings several columns T ij into existence. Each of these columns is fired into the MLP to yield location-aware features f ij . The loss is collected for all locations and backpropagation then estimates the gradients required to update the parameters of the APM. In this entire process, there were no labels being used. The feature grids could have been any layer of a net like DINOv2. APM thus manages to learn a perception field within itself [34] . It can be then frozen, and used as a computational equivalent of any feature-extractor for a downstream-perception task.Training APM in a self-supervised manner using a teacher U.Data: Input data X, Student SAP M , θAP M , Teacher U (frozen), Learning Rate η for each epoch do foreach data point x k in X do f k ← U (x k ) ; T CNN ← x k (create trigger col)3.2, x k ∈ R 3×H×W ; /* Route sample x k in column T [34] */ Tij Unfold ← T 3.4, 1 ≤ i ≤ H, 1 ≤ j ≤ W ; /* Create location-specific queries[34] */ for i in range(H) do for j in range(W) do f ′ ij ← SAP M (Tij) ; /* Feed-Forward column Tij into APM and decode f ′ ij [34] */ RGB ′ ij ← RGB Head (SAP M (Tij))5.1; /* Decode lower-level RGB ′ ij [34] */ L f = L2(f ′ ij , f kij ); LRGB = L2(RGB ′ ij , x kij ); L = L f + LRGB; Compute ∇ θ AP M L; θAP M ← θAP M -η∇ θ AP M L; end end T fold ← Tij3.4 ;/* Collapse all location-specific queries Tij into T [34] */ end end Furthermore, we present the pseudo-code of APM for test-time-training in Algorithm2. First, the textual encoder of the teacher is used for estimating representations of each ground truth class. Then over multiple ttt-iterations, the predicted feature of APM, a.ka. f is refined via statistical running-average 10 . During each learning iteration, the trigger column T is undergoing phases of folding-unfolding. Finally f is being used to perform zero-shot-classification with prior-computed representations R gt .A question may be posed on how to decide the optimal number of ttt iterations t to achieve optimal performance. Indeed, one might build additional inductive-bias in a student (aka APM) to estimate when its own fantasies [46] /predicted semantic-features are better than the teacher's and stop dynamically/recurse until kickoff. Notions on soft decision-making for higher-level cognition are subtly embedded in [101, 3] . ′ ij ← SAP M (Tij) ; /* Feed-Forward column Tij into APM [34] */ f ′ ← statisticalRunningAverage(f ′ ij ) ; Estimate Loss L+ = L2(f ′ ij , f kij ); end end Compute ∇ θ AP M L; θAP M ← θAP M -η∇ θ AP M L;In order to ensure the reproducibility of our experiments, we have shared the model in supplementary during review process. The code, model weights shall be released post-review. APM can work with a single GPU like pascal in less than 2 GB of memory. It can also parallelize on a cluster containing 2 nodes of 8 A6000 amperes. We have provided details of hyper-parameters used in test-time-training (Tab7), and precise details of each layer of APM(Tab 6).Architecture: We inflate APM's architecture in Table6. For the TTT experiments, APM consists of only a single convolutional layer, and 5 MLP layers. Additionally, APM consists of a feature projection head containing a single linear layer, and an optional RGB head. It maybe noted that the number of kernels in the convolutional filter is only 1. This creates a subtle issue: the RGB reconstruction in Fig4 is black and white. This is because a single kernel loses RGB channel information. Put simply, assume a tuple of 3 numbers representing RGB values, < 1, 2, 3 > ,< 4, 1, 1 >. For a convolution operation with a single kernel assuming unit weights, the answer is 6. If this 6 gets injected in the net, it is equally certain that the input was < 1, 2, 3 > or a < 4, 1, 1 > making reconstruction from the RGB head of APM difficult. We found that this symmetry-breaking issue could be noticeably fixed with n kernels ≥ n channels , where n channels is the number of channels c in the input. Historically, various other rotational/translational/mirror-symmetries have played an important role in designing boltzmann machines [88] . The architecture in Table 6 is then meant to showcase APM's potential to an extrema: how much can it do even with a single convolutional filter? Hyperparameters: All hyper-parameters utilized for APM during test-time-training are detailed in 7. We leveraged the seed 42 in most of our experiments, and also conducted experiments with multiple seeds. The weight matrices in APM were initilized with from a random distribution with µ = 0 and σ = 0.01. All the code has been written in Pytorch version 1.13.0. We also note that performing CLIP is a zero-shot model from OpenAI which contains a vision encoder, and a textual encoder. The textual encoder tokenises input class names to features. Both image/text encoder project them to common dimensionality, and classification happens by measuring distances in contrastive space, thereby offering a higher degree of freedom, as opposed to training a class-sensitive linear-probe. CLIP VIT-L features an output CLS token of 768 dimensions, while CLIP VIT-H outputs 1024 dimensions both of which have been accommodated in Tab6. DinoV2 [73] is a foundational-model trained via SSL-objectives and predicts significant semantically-aware representations, which are widely used in various downstream computer-vision tasks.Inductively, both CLIP/Dinov2 rely on VIT, which operates on the principle of parallel attention [100] : image-tokens a.k.a patches can flow along parallel paths among different layers stacked over each other without any loss of spatial-resolution which was also a key-shortcoming of convolutional-nets. Even though the transformers have no bottleneck issue, the attention-operation in each layer still occupies a significant amount of memory [70] .To evaluate model's robustness to distribution shifts, it is necessary to test them on datasets which contain corruptions on a variety of scenarios for eg, fog, rain, snow, etc.. One standard practice has been to take the test set of larger datasets like ImageNet, and create synthetic corruptions to establish appropriate-benchmarks on which the performance of these models can be compared. Alternatively, certain test splits have been manually-curated from-scratch over the internet, for eg, sketches, drawings etc. Below, we detail some of the splits which were used in this paper for APM's robustness experiments.Cifar-10C: is a test-split consisting of 10000 test-samples of Cifar-10, corrupted with 15 noises, across 5 levels of noise-severities. In this paper, we have shown results on the highest level, a.k.a level 5 owing to the resource-constraints.Imagenet-C: ImageNet-C is a dataset split for recognizing objects under distribution shifts, with 1000 classes like original Imagenet. This split contains 15 types of corruptions, with each type containing 5 level of noise severities, aka, the percentage of the image region which is being typically impacted by the corruption.ImageNet-V2: is an independent test set containing images sampled from naturally occuring scenarios, with 10000 images of 1000 ImageNet classes. ImageNet-V2 typically consists of 3 splits, with varying levels of difficulties.ImageNet-A: refers to a curated test-set containing "natural occurring adversarial samples", which were misclassified by the Resnet-50. This particular split contains 7500 images of 200 ImageNet categories.ImageNet-R: refers to a novel test-set of several Imagenet categories, which contain artistic renditions. There are 30000 images in this split spanning across 200 ImageNet categories.ImageNet-Sketch: is a challenging test-split which consists of only black-and-white sketches of 1000 ImageNet categories. This split originally consists of 50,000 images in total.Typically, methods like CLIP evaluate on these ImageNet using a prompt-ensemble of 80 handcrafted templates. APM results were also shown using this ensemble for fair comparisons.Here we perform some additional ablations to understand how varying one of the parameters of APM, while holding others hyper-parameters constant impacts the performance during test-time-training.We perform varying number of ttt iterations, evaluate the performance of APM on three seeds, and report the mean and standard deviations. We note that at t = 25, APM obtains 49.5% accuracy with a minimum observed standard-deviation of 0.2. Effect of varying number of parameters: In Tab9, we perform an additional ablation: the number of parameters inside APM's linear layers are gradually changed from 7M to 120M. We perform TTT iterations on the DTD dataset. As can be observed, the top-1 classification-accuracy of APM increases from 47.0 to 49.1, thereby indicating that a 53M net was optimal for this particular instance of the problem. Beyond that, we observe a gradual drop in the performance, thereby indicating that the net has started to overfit. In an ideal scenario, we would want that lower number of parameters should yield higher performance. However, this would then require some more fundamental changes which allows the net to achieve higher-bandwidths, which remains a matter for the future work [33] . Effect of removing the teacher from APM: APM can perform test-time-training on a single testsample by relying on a CLS-token distilled from a teacher trained on-scale. This might lend itself to the assumption that APM really requires a teacher in order to learn semantically-aware representations.In this ablation, we remove the teacher entirely and have APM perform RGB-reconstruction on COCO-train set. The L 2 RGB-reconstruction loss on COCO-val loss then falls to 0.0027. Training took far longer than if last-layer feature-vectors were also distilled from a teacher into APM.Higher-dimensional vector-spaces carry more bits [89] , but incur significant randomness [5] .Island/vector-distillation speeds-up the learning-process which otherwise might only be informed via RGB-reconstruction and take a lot of time [26, 34] . This helps guide APM's ship to correct points in the subspace. Progressing from VIT b->h in Tab1, shows APM becomes more competitive thereby validating this intuition.Results on Cifar-10C: In Tab10, we show additional results on Cifar 10-C dataset widely used in test-time-training [92] . Cifar-10C consists of 15 types of noise corruptions for all the 10,000 samples present in the Cifar-10 dataset. As evident from the table, CLIP is not naturally robust on Cifar-10 and achieves an error rate as high as 24.5%. This is worse than existing ttt-methods like TTT-Online and UDA-SS. Therefore, CLIP VIT-L is not naturally-robust on Cifar-10c.Using it as a teacher, we get the lowest average error rate of 14.8%, thereby even improving upon the performance of CLIP VIT-L/14. Note that our APM is reinitialized with random weights after everytest sample in contrast to methods like TTT-Online which retain the weights after every ttt-iteration. Inspite of that, we get a lower error rate i.e. APM proposes two technical ideas. 1) The first idea is the proposed column representation T 2) The second idea is the folding-unfolding mechanism. However, there are several deeper non-technical/nonscientific inspirations which motivated the design of APM. We discuss some of those, to help facilitate a deeper-connection and ground our intuitions.A biological analogy [34] : Consider how an organism starts its existence from a cell. The cell is copied across different body locations. Each location possesses identical DNA. However, depending on the location, the DNA decides whether to form an eye or nose. We term this process as unfolding, i.e. a cell 'expands' to yield an organism. Next, there is evidence of jellyfish like Turritopsis dohrnii reverting from their fully grown form to younger polyp states [77] . We term this process as folding, i.e. cells of an organism collapse back to the single cell it began from.A computational analogy [34] : We now start treating an image I as a digital organism. It starts from some compressed representation T . T unfolds to yield the image I. I then folds back to yield the compressed representation T . Learning proceeds by oscillating between these unfolded and folded phases. At every step, the net is trying to reconstruct image I from T . T is then expected to be a dense vector-space.A cellular-automaton analogy [106] : On surface it seems a pretty trivial matter to discuss: a point can expand and yield beautiful patterns which can either be an entire universe in accordance with the theory of big-bang, or can be reproduction of an organism from a singular zygote. But, it is funny: if you start from a point, and unfold it, then all you can get is a sphere. This appears to be true for the behaviour of light, in accordance to Huygens principle 11 . However, we observe non-spherical objects around us all the time. Turing posited that the symmetry breaking in the sphere must happen somewhere while the organism unfolds: such patterns could then be explained a variety of diffusion based equations [48] .Figure 7 : Cifar 10 islands: Individual part-wholes are clearly observed in APM features. These features are used for downstream classification.We leverage the visualization mechanism by [70] . These islands are not been hand-picked [34] This idea has been explored in cellular automaton: different replication rules of starting point can yield different final patterns 12 . Scientists then continue to derive different rules which yield different patterns, which is akin to how we were resorting to hand-engineering features in deep-learning for a long time. APM attempts to answer the question: Is it possible to build a learning machine which can start from a point, unfold, and then express correct features at the correct place?. We want to then push the job of rule-learning to what backpropogation does best. We have lost the "why" for the knowledge was encoded in the weights of the net, but we seem to have gained the ability of correct features presenting themselves at correct locations. This location-aware-disentanglement procedure thereby represents a step towards solving Arnold's superposition theoram and Hilbert's thirteenth problem [34] . However, backpropogation can only approximate solutions and not yet reach exact ones [33] , and mathematical formulations are lost into the weights of the neural-net.We then begin to imagine learning machines [97, 98, 96] which can solve a complex problem like cryptography/breaking-a-cipher in two phases 1) relax the system towards an approximate solution [45] 2) have the system spit-out which parts of the solution are uncertain, and brute-force towards the remaining solution. Or, we could make the loss of the learning-machine reach perfectly zero, thereby representing a perfect solution 13 . Hard problems like recognizing faces are approximately solved as a consequence of a single forward pass through a learning machine. If the loss could be made to reach Our APM is trained on COCO-train set, and islands on COCO-val set are visualized [34] . We leverage the visualization mechanism by [70] . Note that these islands are a consequence of single feed-forward through the net and not an iterative routing mechanism as in [18] . These islands are not been hand-picked. [34] zero, then we could consider the problem to be perfectly solved. Solvability can then happen in a feed-forward phase, which for practical purposes appears to be polynomial. 14 Next, we redirect the reader's attention to neumann's theory of self-reproducing automaton [72] . His idea of a self-replicating colony was that there is an infinite source of resources a.ka. reservoir which is shared by the automaton which operate at different locations of a colony. The colony uses up the shared resources, does self-replication and in this way converts raw materials/matter into useful intelligent-behaviour. The infinite reservoir of machines he talks about then reduces to the trigger column T in APM: since features are sampled from a same space, they automatically become aware have been some old ideas to reduce this degree of freedom. We would then need a mechanism to dynamically adjust the degree of freedom during inference. 14 A computing machine which is exponentially fast can pass on the illusion that a non-polynomial solutionspace is now polynomial, for the time is measured in a constant observer-space [13] . However, feed-forward loses how the solution was reached, i.e. we can no longer predict precise algorithmic steps. Mathematical-imprecision could still lead to a perfect solution. An extreme loss of precision means that the net could then run in an analog hardware [33] of themselves, thereby making explicit attention unnecessary. This also then is same as how latents have been classically sampled in the generative models [22] . One might argue that multiple automaton although starting their lives at the same point will need to communicate among themselves, as they differ among their configurations at later point in their lifespans 15 . Fluctuations in T are then akin to mutations. We compensate for this fact by weight-sharing the MLP across different locations in APM.A cosmological analogy: In physics, one of the famous theories of the origin of universe has been starting from a single point, and undergoing a continuous expansion [50] . There are alternate theories for eg, Conformal Cyclic Cosmology [75] which hypothesize the universe undergoing periodic cycles of expansion and contraction [79] . Drawing inspiration from these fundamental insights, the trigger column T undergoes these cycles of folding and unfolding during the learning iterations.I A new representation: Hinton's Islands of agreement [70, 34] APM is inspired from GLOM's philosophy. This section explains the principles behind island of agreement in more detail to facilitate an easier understanding. GLOM [34] assumes that each pixel of an input image contains a higher dimensional column vector over it. Therefore, for an image of h × w, there is a column vector at every location. Next, all these column vectors undergo some sort of message passing between themselves, for eg, via attention. At the end of these procedure, the idea is that the vectors belonging to identical objects should start pointing in the same directions in the higher dimensional embedding space. A cluster of such higher dimensional vectors is then called islands of agreement. To see such islands in practice, one can then apply a dimensionality reduction procedure like t-sne to 3 dimensions, and visualize the obtained feature map by backprojecting the obtained features to the RGB range of [0,255] [70] . Note that t-sNE preserves the higher dimensional spatial structure among the vectors since it fits gaussians. This visualization mechanism does not require any feed-forwarding or backpropogation through the net: the islands are already there, and clustering merely helps them to reveal their presence 16 .We present the islands of agreement which can be observed now in APM in Fig 7, 10, 11 . Notice how the part-wholes in the images are clearly observable in different colours. Finally, we scaled up our APM and trained it on COCO train set. Fig 13, 12, 8, 14 visualizes the islands of agreement on COCO val set. It can clearly be seen that the net develops a semantic clustering and shows the potential to do a dense task. APM offers a unique advantage over parallel perception (DINOv2): features at a particular location can be queried serially. One can 'selectively' query the locations which correspond to a particular object [109] , instead of bringing the whole feature grid in existence and then choosing the relevant objects [6] . Such an inductive bias of choosing which of the input rays/columns in a neural field corresponds to which object has already been explored by Yu et al [109] .In the presented islands of agreement, one can see that the parts and wholes of the object are all entangled into one image. In an idealistic scenario, the net should learn to map the whole <part, whole , relationship> triplet [34] . GLOM looks at the features predicted by different layers of a transformer in a different way: lower layers are predicting object parts, and higher layers are predict the full object. For now, APM has only modelled the last layer of the transformer. In an idealistic scenario, we want the net to traverse the whole up-down part-whole hierarchy as well as learn the pose-transformation matrix which can get us from one part to its whole [34] .Let us next consider a 2D image of a man [8] gazing at the ground in front of him. At the object level it does not make sense for embeddings of the nose to jump to the embeddings of the ground. However, it is okay to learn pose-transformation matrices which can get us from one point on the man's body to another. This means that this restricted movement has to be informed by a top-down feedback which we have not yet modelled. Furthermore, this pose prediction can happen by a big fat-net like APM 17shared across locations and levels of the part whole hierarchy. This is different from capsules, which contain only a few convolution filters at a particular location. In computer graphics, the pose matrix is defined a four by four matrix where the first three by three elements are rotation components, last row are homogenous coordinates, and first three numbers in the last column are camera translations. If the net predicts a 4 × 4 matrix each of whose element can be any constrained number, we lose the ability to make interpretations about what the predicted matrix actually represents since it is no longer in a well defined world-coordinate frame. This problem is same as how the 'learnt' camera poses are aligned to 'real' camera poses in neural fields while doing bundle adjustment [54] .Note that islands of agreement operate at the finest spatial granularity: there is one column vector for each pixel of the input percept. However, in the case of perceptual overcrowding [47] , there might be some hidden part of an object whose inference might be made seeing the visible pixels around it. However, the island in our case does not model this, since agreement is only established for one object at a particular location [70] 18 . Quod errat demonstrandum. Q.E.D [15] . May these humble neural-nets [66] now fold themselves [31] and stay safe [4] . For all of us. The last section of forward-forward paper talks about Mortal computation and it now appears that we might reach that point [33] . AI safety however still needs a lot more work [27] .Till the next time the space-curves [13] , little_dogzilla aboard 'Starship Connectionist Enterprise'. End of transmission.Figure 9 : A cute-little-godzilla trying its best to navigate wormholes [104] . Hiyaaa!! 18 We could use non-intuitive hungarian-matching to resolve order among predictions of multiple object in the scene [70, 6] . It is non-intuitive, for the objects in this material-world don't permute in reality, and dont choose the combination which shall result in the lowest loss. Rather the brain surfs reality [34] . The brain then is the neural net, and the objects are the external input from the environment [45] . The objects/islands are already there, and reveal themselves when fixated upon. If not, they then just collapse into a superimposed quantum state, for a tilted cube or a diamond are nothing but a same object [76] . The granularity of fixation can then vary in practice,aka saccades. Quantum state can then arise in a digital machine as a consequence of kolmogorov's theoram. Such effects can then happen at room temperature instead of super-cooled matter states [14] . The only difference is that a system of parallel-connected components requires gradient descent in a digital machine. In a quantum machine, there can be a different mechanism for relaxation. These features are used for downstream classification. We leverage the visualization mechanism by [70] . These islands are not been hand-picked. [34] NeurIPS Paper Checklist Justification: The first claim is that we improve on ttt's computational performance, second claim is that APM performs better than existing baselines, third claim is that APM's architecture is a single CNN filter and a MLP of 5 layers. Fourth claim, to the best of our knowledge is that we are providing the first evidence of GLOM's insight: i.e. percept is really a field . The first claim of computational complexity is analyzed in Tab 4 and Fig 2 . Second claim of improved performance over ttt methods is present in tables 132.Third claim of APM's architecture will be evident in the codebase shared with this manuscript. Final claim confirming GLOM's insight i.e. percept really is a field is evident in Fig6. A kind These features are used for downstream classification.We leverage the visualization mechanism by [70] . These islands are not been hand-picked. [34] reader can also interpolate any two images in the wild using the codebase we shall share with this manuscript in the review process.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.A flick of the eye, the fall of her hair, the passing of a second, or capturing the world at 120fps: instantaneous can be a subjective experience. Seconds can sometimes pass slowly for other reasons also.The word triggering has been chosen because this column shall be used to trigger the queries to the neural field shared across different locations[34].It means generating a unique representation Tij for any image and a location selected in it. Another way to break symmetry is to add noise[45]. Noise also helps escape degenerate local-minima. These ideas can be traced back to boltzmann-machines/hopfield-nets[49].A larger number of TTT iterations are needed to perceive semantically-meaningful islands. Actual TTT process is far quicker, it takes around 20 iterations.This particular net got a L2 grid loss of 0.15.Thku thku so much for tracks for babies like us[23].It can be decentralized.They are not always sleeping, they sometimes wake up too.[46].Averaging is similar to pooling in convolutional-nets and might lose important information over time. It might be helpful to explore temporally-weighed averages or alternate type of long-term memory-banks[107].With a point on the wavefront being its own source. But the envelope is still a growing sphere.Interaction is only between local neighbours, yet order appears. Local-interaction is a form of local-routing, and somehow stable patterns appear without a global constraint. Unfortunately, backpropogation still relies on a global constraint. However, that symmetry was recently broken[60]. Perhaps, the answer then lies in how the starting point instructs the unfolding process[29].If the degree of freedom of the machine ≥ than the freedom needed to solve a given problem, the machine just keeps interpolating in the subspace among multiple set of weights that give the same answer[37]. This wastes computation even if the solution has been reached. Annealing, early-stopping and quantizing weightsAs a consequence of inherent markov-stochasticity. Biologically, Darwin called it evolution[10]. In machines, it can be inbuilt randomness in a computing element[98,97,96].In this material nature, the boundaries around objects are not precise boxes. Rather they gradually blend in the background. This is known as sfmato effect which leonardo hath incorporated whilst painting Mona Lisa[67]. Similarly, GLOM proposes to do away with boxes altogether[34,70].A big-fat net being used to predict embedding at each location in the part-whole hierarchy is different from simply making only the embeddings compete among themselves[18]. Replicating the same net across locations is costlier than replicating location-specific input-queries to a shared net. Both achieve the same effect. Of course, our weight sharing is not biologically plausible, which also remains a deficiency of GLOM[34]. 


Decision: The paper proposes a novel test-time training method, Asynchronous Perception Machines (APMs), which is computationally efficient and demonstrates robust performance on out-of-distribution image classification tasks. The method leverages patch-based processing and knowledge distillation to improve classification accuracy while requiring fewer computational resources. Strengths include its novel approach to handling test-time training and the clear empirical validation of its effectiveness. The method’s ability to process image patches in any order and its competitive performance across benchmarks are notable. Please address reviewers’ comments in the final version.

Reviews: review: paper_summary: paper_summary: This paper proposes a computationally-efficient architecture for test-time-training. APM can process patches of an image in any order asymmetrically, where it learns using single representation and starts predicting semantically-aware features. The experiment results demonstrates the effectiveness and efficiency of the method.
strength_weakness: strength_weakness: 1. The writing can be improved. The current organization makes me feel very confused of the proposed method. For example, in the abstract, the novelty should be emphasized. It seems that processing patches of an image one at a time in any order asymmetrically is emphasized, which I don't think it is a novel contribution.

2. As this paper targets at solving problems in test-time-training, I think more related work in this domain should be introduced in the section of related work.

3. I cannot understand why the analogy compared with biology and computation makes sense. The authors introduced a lot of analogy, which has no proof and no explanation. I think the technical soundness of the paper should be significantly improved.

4. The training and test details in the experimental section should be demonstrated in details, as readers may be interested in the experimental setting.

5. I am wondering if the proposed method can be extended to few-shot testing scenarios?
questions: questions: See weakness.
limitations: limitations: See weakness.
paper_summary: paper_summary: The authors propose a new test time training method (architecture + self-supervised task) called Asynchronous Perception Machines (APMs). APM is computationally efficient, and empirically matches or improves performance on OOD image classification tasks versus prior test time inference approaches. The approach considers each $c' \times h \times w$ feature vector in the output of an image encoder as a c' dimensional mapping of the input spatial features to this 'island of agreement'. A convolutional neural network is used to generate this from the image. Once generated, this location aware column is folded/unfolded at inference time based on the input patch, and used to reconstruct the RGB and location specific features. The features are then averaged and compared against a text feature vector to perform classification. APM processes image patches one at a time in any order, thus also exploiting the benefits of transformer style models. Empirical results show that APM does well on out-of-distribution image classification benchmarks and exhibits competitive performance compared to existing TTT.
strength_weakness: strength_weakness: - While I think test time training is an important and useful paradigm, the illustrative example of "self-driving car trying to make sense of the visual world while it is raining, but there were no such training scenarios in its training data" (L20-21) does not really apply because tts does not really allow for instantaneous decision on a new test instance.
- I don't think the analogies in 3.1 are anything beyond that, and as such don't belong in the main paper. It would be better to draw real scientific analogies in the main paper, e.g. to 3D novel view synthesis.
questions: questions: Typo L17: Neural can now -> Neural nets can now
limitations: limitations: Discussed briefly in Section 8
paper_summary: paper_summary: This paper introduces a new algorithm for test-time training where the “test-time” task is overfitting for per-image CLIP / DINO feature distillation. The associated “downstream” task involves directly using the per-image network feature to perform image classification using dot product in CLIP space. Given an image, the method first proposes encoding it into a global feature vector via convolution. The global feature is then unfolded into per-pixel feature vectors by concatenating positional codes to the global features. These local features are then decoded and aggregated before being supervised with the CLIP image features distillation loss. This process is repeated for multiple iterations, as the global feature is “refined”. The final global feature turns out to be competitive or better than the base teacher model for some tasks and outperforms baselines. Despite the global CLS-token only supervision, the local bottleneck learns useful dense semantic features which are qualitatively visualized. The paper also shows some interesting qualitative findings when the method is trained across a dataset instead of on a per-image basis.
strength_weakness: strength_weakness: I believe that the paper would benefit from some re-organization, clarification and writing improvements. At different points in the paper, there seems to be contradictory information regarding the nature of the method and some key details necessary to the success of the method are left out. Bulleted below: 

-	In the method figure (Figure 1), a flattened rgb patch seems to be appended to the flattened image encoded with convolution, annotated $I_{xy}$. However, no references to this variable are made in the methods section. Further the methods section only mentions concatenating the global feature with the positional codes, without appending any patch specific information while querying the field. 
-	In the same figure (Figure 1), the “folded state” in the yellow box seems to be the one that is fed into the MLP for decoding. However, in the methods section, it is claimed that it is the unfolded state that is sequentially fed into the MLP for decoding. 
-	In the figure and on lines 65, 129, the method makes a reference to auto-regressive querying. I’m not sure what this means, as any way in which the model keeps “state” or “memory” is not clearly described. Further, it is not described how the output of any one MLP call is fed back into future steps. After looking at the appendix pseudocode, there does not seem to be any auto-regressive component. Was the intention to mention just sequential (and not auto-regressive) encoding?
-	Details about the unfolding process are clearly described, however details about the folding process are scarce. In Figure 1, the folding process seems to include a gather operation, while elsewhere it is described as an averaging operation. Further, the appendix pseudocode is lacking the “multi-iteration” nature of the algorithm. 
-	When additional patch level feature losses are added (line 262) in the later sections, the details for how these are supervised are not clearly mentioned in writing. 
-	The related work section needs to be significantly improved: it is presently under-cited and leaves out key related work in test-time training and prompting. These are mentioned and the differences described in other parts of the paper – but reorganization is required to improve readability. A further line of related work is in feature refinement / upscaling / distillation, and such papers need to be cited and compared to in order to elucidate differences and potential novelty in distillation method. 

The paper contains several unsubstantiated claims that weaken the otherwise interesting results. Perhaps, they can be clarified better, or removed:

-	The paper claims that APM confirms that “percept is a field” as speculated by GLOM. While the paper does show that field-based decoding improves image classification, and that local features are emergent from global features, I believe a more general purpose evaluation of a broader range of perception tasks, including dense ones, must be done before this insight is claimed. As such, I believe this statement needs clarification as to what subset of ideas of GLOM are actually being “shown to work” in this paper. 
-	The paper claims that APM doesn’t need dataset specific pre-training (Eg. line 176). While this is true in the sense that APM is a test-time instance method, APM does require access to highly meaningful visual representations (Eg. DINO / CLIP) that have been trained at scale. APM performance, as shown by the tables, degrades as the quality of the target representation degrades. I believe paraphrasing this claim is important to reflect this.
questions: questions: I have several questions. Mainly important to clarification of the method:

-	APM compares to baselines such as Prompt ensembling in Table 1, which is a text query side improvement. Did authors similarly use the best performing “text prompts” for their method?

-	The per-instance distillation loss proposed by APM is minimized when the predicted feature is exactly the same as teacher feature. The fact that the final feature surpasses teacher feature performance implies that the loss does not go to 0 at training. Intuitively, what is the key constraint in the network that prevents loss from going to 0? What do the training curves look like for some example runs? 

-	Summarizing the questions asked in weakness section 1:

o	Could you clarify what exactly does the folding process look like? 
o	Is $I_xy$ (patch level info) appended to $T$ and fed into the decoding MLP? 
o	Is the model actually auto-regressive or purely sequential / iterative?
o	What is the correct pseudo-code corrected with the 20 iterations? How is T (the encoded global feature) updated after one iteration of encoding?
o	How does performance on classification degrade with fewer iterations?

Answers to these questions are important for me to fully understand the paper.
limitations: limitations: The limitations section as currently written is a future work section. As mentioned in the “questions” section above, dense tasks and utility of local features is indeed interesting as a next direction. Additional things that can be discussed in the limitations section is the need for the method to do multiple iterations of test-time training instead of simple feed-forward inference or a single gradient step like other test-time training methods, and the variability in performance as compared to baselines.
paper_summary: paper_summary: The authors propose a novel test-time training method called the Asynchronous Perception Machine. This method leverages knowledge distillation from other pretrained networks, such as CLIP. The main contributions of this approach are its robust accuracy in classifying corrupted images and its computational efficiency, requiring fewer FLOPs compared to other models.
strength_weakness: strength_weakness: 1. Lack of Related Works: The paper proposes a novel test-time training method but references fewer than five related works. However, there are more than 50 papers on test-time adaptation methods published in recent years (e.g., [1-8]), none of which are mentioned.
2. Lack of Comparison to State-of-the-Art Methods: The manuscript identifies TTT-MAE as the second-best method, yet this method was introduced in 2022. Since then, many methods, such as Diffusion-TTA [4] and DeYo [8], have surpassed TTT-MAE.
3. Insufficient Explanation of Proposed Method: The proposed method is not explained in sufficient detail. For instance, the Vision Encoder is only depicted in a figure and not discussed further in the text.
4. Misleading Information The “Trained” column in one of the tables is misleading. TTT-MAE does not require training data during the test phase; it requires an auxiliary loss during training.
5. Inadequate Ablation Study: The ablation study lacks relevance. While the authors propose adaptation to distribution shifts (i.e., corrupted images), the ablation is conducted only on CIFAR-10 and CIFAR-100, which are datasets of clean images.

Reference

[1] Tent: Fully Test-time Adaptation by Entropy Minimization

[2] Continual Test-Time Domain Adaptation

[3] Robust Test-Time Adaptation in Dynamic Scenarios

[4] Diffusion-TTA: Test-time Adaptation of Discriminative Models via Generative Feedback 

[5] NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation

[6] Robust Test-Time Adaptation in Dynamic Scenarios

[7] SoTTA: Robust Test-Time Adaptation on Noisy Data Streams

[8] Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors
questions: questions: It is very unclear what component in the Fig 1 is. Specifically, what computations do they do?
limitations: limitations: There is no concerned limitation.

Comments: title: Author's response to the Respected Reviewer doB6
Response to Rebuttal
Author's reply to the respected reviewer KZWi
Author's response to the Respected Reviewer X1eH
Author's reply to the respected Reviewer PHih.
Author's reply to the respected Reviewer PHih.
[Authors Reply] List of references used in our rebuttal to the respected Reviewer PHih
[Authors Reply] List of references used in our rebuttal to the respected Reviewer KZWi
comment: We thank the respected reviewer doB6 for their kind words with regards to APM. We will further improve the paper's clarity and organization for better engagement with a prospective reader.

We remain, 

Yours sincerely,

Authors.
I'd like to thank the authors for taking the time to respond to the review, as well as for accepting some of my suggestions regarding the paper. After reading through the general rebuttal, as well as the author's rebuttals to each of the reviews - I agree with other reviewers that the writing can be improved (especially in terms of clarity and organization). This can be done in the camera ready, and as such I think the APM paper is good enough to appear at NeurIPS. I would like to keep my original rating of Accept, and look forward to seeing this work at NeurIPS.
We thank the respected reviewer KZWi for their kind consideration of our work. As promised, we will incorporate the writing comments and the reviewer feedback to improve the engagement of a prospective reader.
Thank you so much for your valuable time, and we will be grateful to respond to any further clarifications,

Yours sincerely,

Authors.
Thank you for your rebuttal. My main concern with the paper (reason for borderline score) was the lack of clarity regarding the technical details of the folding / unfolding . distillation processes  and the lack of clarification / phrasing of certain claims regarding inspirations / analogies in the paper. The authors have sufficiently addressed the technical questions, clarified intuition and promised to incorporate the writing changes in the final draft of the paper. These promised changes would be important for a strong final paper. 

With my questions clarified, I will increase my score.
We thank the respected reviewer for their timely response to our rebuttal. We are grateful for the reviewer's consideration and will be happy to respond to further clarifications.

Yours sincerely, 

Authors.
Thanks for your rebuttal. Most of my concerns are well addressed, so I increased my score.
We thank the respected reviewer for their valuable time, and efforts in helping us to improve our work. We are grateful for the chance to engage deeply in the inner workings of APM.

In our reply to the respected reviewer Kzwl ```On memory/states and auto-regression in APM```, we mentioned substituting the word autoregression with sequential, because the trigger column $T$ unfolds to yield location aware columns, i.e. $T_{ij} = (T|p_{ij})$, where $p_{ij}$ is the generated hard-coded positional encoding as being used in transformers[3], and neural fields[2]. These $T_{ij}$ columns are responsible for sequential querying of the MLP.

Mathematically, the MLP predicts a location specific feature $f_{ij}=MLP(T_{ij})$ as a consequence of a forward-pass as each column gets pumped through the net. The MLP is queried ```sequentially``` with different columns $T_{ij}$ until the locations $ 1 \leq i \leq H$, $1 \leq j \leq W$ get exhausted, where $H,W$ are the dimensions of the input image $I$.

In the shared code in supplemental, MLP does not contain explicit recurrence, because the trigger column $T_{ij}$ carries the **entire** sequence $T$. The $p_{ij}$ in $T_{ij}$ guides the MLP to decode location-specific feature $f_{ij}$, which is a form of feature-expression as also previously hypothesized in GLOM [1] (our changes over Section 2.1 and Figure 3 in the GLOM paper).

We can also gain more depth by looking at Fig 3 of the GLOM [1]. The subtle differences are, 1) [a b] in that figure is the trigger column $T$ in APM, which carries the whole image $I$. 2)$x4$ is substituted by a positional encoding $p_{ij}$. 3) In addition to decoding location-specific RGB, we also decode higher dimensional features $f_{ij}$, for $ 1 \leq i \leq H$, $1 \leq j \leq W$ which allows APM to do field-based-decoding, which we leveraged for downstream-classification.
We have also added a detailed pseudo-code for operation of APM during test-time-training in
[Pseudocode](https://anonymous.4open.science/r/apm_rebuttal-F3D1/ttt_pseudocode.png).

The behaviour of APM is then akin to neural fields[2] (as suggested by the respected reviewer doB6), for eg, neural fields fire iid rays into the MLP shared across all input rays and decode location-specific rgb. Similarly, APM shares a MLP across all columns and decodes location-specific features. While neural fields work for 3D view synthesis, APM works for 2D image perception, with potential for extensions to other higher-dimensional input-percept, as also duly noted in Sec 11 of the GLOM paper[1].

As promised, we will also refine the paper-writing for better engagement with a prospective reader.

Yours sincerely,

Authors.

[1] Hinton, Geoffrey. "How to represent part-whole hierarchies in a neural network." Neural Computation 35.3 (2023): 413-452.

[2] Mildenhall, Ben, et al. "Nerf: Representing scenes as neural radiance fields for view synthesis." Communications of the ACM 65.1 (2021): 99-106.

[3] Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attention is all you need. Advances in neural information processing systems. 2017;30.
Thank you for the clarification. My concern has been addressed and I raised the score. This can be a good and interesting work after polishing writing for a few iterations.

I have one more question, in the paper, you mention "Therefore, the MLP can be queried autoregressively."
What is the mathematical formulation of the autoregression you mentioned? Does the MLP produce the output based on the previous T?
We thank the reviewer for their timely response to our rebuttal. We are grateful for the reviewer's insight and help in improving our work.

We agree with the reviewer that the differences between TTA/TTT are indeed subtle/minor. As this difference do not play an important role in our contributions, we will refrain from differentiating between TTT/TTA works in our paper, and add more TTA works in related work and comparisons. We have shared comparisons with the recent TTA work aka Diffusion-TTA, and we will also add other such works.

In our work, we have followed the experimental setup of TTT-MAE [NeurIPS 2022] and TPT [NeurIPS 2022]. Furthermore, we will be grateful to mention the reference to TENT [Neurips 2020] in TTT-MAE paper: "Other papers following [42] have worked on related but different problem settings, assuming access to an entire dataset (e.g. TTT++ [28 ]) or batch (e.g. TENT [48 ]) of test inputs from the same distribution. **Our paper does not make such assumptions, and evaluates on each single test sample independently**".

Finally, we also point towards mention of TENT in the TPT paper: "However, TENT **needs more than one test sample** to get a non-trivial solution."

We would like to clarify a few other points:

- APM does not rely on any pretraining since weights are drawn from a normal distribution.
While APM does rely on CLIP-pretrained weights, **we do not perform any training/pretraining on ImageNet**. We would be grateful to mention that CLIP was trained with 400M image-text pairs from OpenAI, and Openclip variant leveraged dfn5b. Furthermore, APM performs competitively on CLIP baselines and other methods which use CLIP weights.

>> "Moreover, "TTT adapts to each sample independently" is completely wrong. It is unnecessary that TTT has to either adapt to one sample or the entire batch or test set"

We agree with the reviewer here that TTT does not have to adapt to a single sample. Existing works like TTT-MAE[1] and others have focused on adapting to a single sample, as noted in their paper "By test-time training on the test inputs independently, we do not assume that they come from the same distribution".

We will be happy to discuss any further concerns,

Yours sincerely,

Authors.

[1] Gandelsman, Yossi, et al. "Test-time training with masked autoencoders." NeurIPS2022.

[2] Shu, Manli, et al. "Test-time prompt tuning for zero-shot generalization in vision-language models." NeurIPS2022

[3] Sun, Yu, et al. "Test-time training with self-supervision for generalization under distribution shifts." International conference on machine learning
While I have no doubt about the performance and efficiency of the APM, I have a serious concern that misunderstanding the terminologies and previous work's experiment settings may affect the reliability of the work. The misunderstanding implies that you never read any of TTT or TTA papers in detail and completely misunderstand their settings. My arguments are as follows:

Response to "TTA adapts to the target distribution (using all test samples), whereas TTT adapts to each sample independently.

This is completely wrong. Most TTA do not adapt to the whole test set. Many of them adapt to the batch samples or even to instance.
Moreover, "TTT adapts to each sample independently" is completely wrong. It is unnecessary that TTT has to either adapt to one sample or the entire batch or test set. TTT/TTA lies on a special case of Domain Adaptation whereas the model has no access to the source data (source-free) but has to adapt to the target data during the test time. 

I suggest you read through all the papers about TTT and TTA. You will see some people in the community use TTA and TTT interchangeably [2]. I can see no significant difference between to the general setting between TTT and TTA.
However, the pioneer like TENT [3] discusses a small difference between TTT and TTA. TTT requires the adjustment of training loss (adding an auxiliary loss term) while TTA only introduces the new loss during the test time.


I can refer to "Network details" on section 3 of the pioneer test-time training (TTT) [1] (which some people also classify it as test-time adaptation [2]). They use different ResNets trained on ImageNet and CIFAR10. ImageNet-trained ResNet is evaluated on ImageNet-C and the same for CIFAR10. Regarding your method, as you use pretrained Vision Encoder as teacher, the vision encoder is already trained on ImageNet. This means your method also uses the knowledge from the training distribution extracted by the teacher model.



[1] Test-Time Training with Self-Supervision for Generalization under Distribution Shifts 

[2] A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts

[3] Tent: Fully Test-time Adaptation by Entropy Minimization
**List of References in Rebuttal**

[R1] Diffusion-TTA: Test-time Adaptation of Discriminative Models via Generative Feedback 

[R2] Continual Test-Time Domain Adaptation

[R3] Robust Test-Time Adaptation in Dynamic Scenarios

[R4] Shu, Manli, et al. "Test-time prompt tuning for zero-shot generalization in vision-language models”.

[R5] NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation

[R6] Robust Test-Time Adaptation in Dynamic Scenarios

[R7] SoTTA: Robust Test-Time Adaptation on Noisy Data Streams

[R8] Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors

[R9] Tent: Fully Test-time Adaptation by Entropy Minimization, ICLR 2021
p.s. other references follow paper order.

Yours Sincerely,

Authors

R1: Einstein, Albert. "On the electrodynamics of moving bodies."

R2: Geoff Hinton at Stanford, YouTube video, timestamp 49:07, https://www.youtube.com/watch?v=CYaju6aCMoQ

R3: LA Homogeneous Universe of Constant Mass .

R4: Neumann, John. "Theory of self-reproducing automata."

R5: Shannon, Claude Elwood. "A mathematical theory of communication."

R6: Hinton, Geoffrey E., et al. "The 'wake-sleep' algorithm for unsupervised neural networks."

R7: Bengio, Yoshua. "The consciousness prior."