VISTA: A Panoramic View of Neural Representations

Authors: tom  white
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-23 12:05:15

Abstract: We present VISTA (Visualization of Internal States and Their Associations), a novel pipeline for visually exploring and interpreting neural network representations. VISTA addresses the challenge of analyzing vast multidimensional spaces in modern machine learning models by mapping representations into a semantic 2D space. The resulting collages visually reveal patterns and relationships within internal representations. We demonstrate VISTA’s utility by applying it to sparse autoencoder latents uncovering new properties and interpretations. We review the VISTA methodology, present findings from our case study, and discuss implications for neural network interpretability across various domains of machine learning.

Main paper: 
 introduction: 
 Deciphering the internal representations of neural networks and biological brains remains a formidable challenge in both artificial intelligence and neuroscience. As models grow in complexity and scale, traditional methods of analysis struggle to provide comprehensive insights into the vast landscape of learned features and concepts. This challenge is further compounded by the emerging evidence that different learning systems, when exposed to similar stimuli, often develop comparable internal representations -a phenomenon that demands novel approaches for investigation and understanding (Sucholutsky et al., 2023) .Building on previous work focusing on visualisations for interpretability (Carter et al., 2019) , we propose VISTA (Visualization of Internal States and Their Associations) which combines clustering with text-to-image models to create a visual "cartography" of neural representations. VISTA offers a unique approach to exploring and interpreting high-dimensional spaces by generating visual collages that map the semantic relationships between different elements of a representation. This method aims to provide researchers with an intuitive, visual interface for navigating the complex terrain of neural representations, potentially revealing patterns and structures that might otherwise remain hidden.As an initial test of VISTA's capabilities, we apply our method to the domain of the latent codes of sparse autoencoders (Cunningham et al., 2023; Bricken et al., 2023) , an area where the current state of the art relies heavily on automated interpretability techniques using large language models (LLMs) (Bills et al., 2023) . We examine three use cases using the Gemma-2B SAE model (Lieberum et al., 2024) , comparing VISTA's findings with those obtained through automated methods. Our results indicate that while VISTA often corroborates the features identified by LLM-based techniques, it also demonstrates the ability to uncover deeper meanings and associations that automated methods currently miss. This suggests that VISTA could serve as a valuable complementary tool in the broader effort to understand and unify representations across different neural models. 
methodology: 
 The VISTA pipeline consists of several steps designed to create an interactive, visual representation of high-dimensional data.1. Representation and Dataset Selection: Choose a dataset of interest and a representation that assigns a feature vector to each input.2. Data Encoding: Encode the dataset using the selected representation.3. Dimensionality Reduction: Apply UMAP clustering to project the encoded data into a 2D space, preserving nearest neighbor relationships from the higher-dimensional representation (McInnes et al., 2020) .4. Cartographic Rendering: Generate a visual map of the 2D space. This process involves dividing the 2D space into smaller tiles, extracting the subset of the original dataset that corresponds to each area of the UMAP projection, and rendering one item from each subset as part of a MultiDiffusion panorama (Bar-Tal et al., 2023) .5. Interactive Visualization: Present the resulting graphic in an interactive interface, allowing exploration of the visual space alongside the corresponding data points.The cartographic rendering has been iteratively refined to highlight semantic structure. The UMAP clusters abstracted into areas of high probability with connections between clusters explicitly shown and overlaid with semantic information from the text-to-image process. Taking advantage of the iterative nature of the diffusion pipeline, we select different inputs within each region at each step of the diffusion process to semantically smooth out the drawing process and remove outliers. Together these produce a more coherent and representative visualization.The final result is an interactive graphic that invites exploration. Visual anomalies naturally draw attention, encouraging closer inspection and potentially revealing unexpected patterns or relationships within the data.Our efforts of understanding shared representational structure are grounded in using mutual nearest neighbor metric for measuring alignment. This is consistent with recent research on representational alignment (Huh et al., 2024) . The fidelity of the resulting VISTA map itself can also be measured directly as nearest neighbor analysis is meaningful across both low and high dimensional spaces. We report these map accuracies as mutual-knn "gain" which explicitly removes the null hypothesis of chance neighbor pairings as k increases (see appendix A for notes).In the following section, we present a specific use case and implementation results to demonstrate the effectiveness of the VISTA pipeline in practice.3 Experiment: Interpreting Sparse Latents Sparse AutoEncoders (SAEs) have emerged as a promising technique in machine interpretability, but understanding the "meaning" encoded in each of the thousands of latents remains elusive. Current best practice is "Automated Interpretability" in which the latents are fed into LLMs and they come up with suggested labels for each of the latents (Bills et al., 2023) .We constructed a VISTA pipeline to explore the latents of the publicly released Gemma2-2B SAE latents (Lieberum et al., 2024) . Here we use the implementation covered above and look at the resulting visualizations for 3 latents in particular to see how our approach is compatible with the automated techniques, but able to surface new and previously hidden properties. 
implementation: 
 Following the steps outlined in our methodology:• Representation and Dataset: We chose Gemma2-2B SAE 16k residual layer 20 as the representation. For the dataset we chose 200,000 auto-generated short captions from the Human Preference Synthetic Dataset (ProGamerGov, 2024) which follows the methodology of Betker et al. (2023) . For each map we take the top 4000 (2%) captions that maximize the selected latent, and each caption is represented by this SAE representation.• Dimensionality Reduction: We applied a custom distanct metric which sums the cosine distance in SAE space with the absolute difference along the latent axis. This was mapped into a UMAP with an asymmetric aspect ratio.• Cartographic Rendering: We used a version of MultiDiffusion running on the Flux.1[dev] diffusion model (Labs, 2024) , with 100 diffusion steps and a minimum of 4 points per cluster for semantic smoothing.• Interactive Visualization: The resulting graphic was placed in an interactive interface for exploration, with the original 4000 captions arranged spatially to preserve nearest neighbors.These choices provided a practical way to make an initial evaluation of our approach. The dataset was selected as it contains a variety of subjects and is already well suited for use in a text-to-image context. The UMAP hyperparameters were intended to allow "stretching" along the latent axis and though this wasn't always achieved as intended we did see stable clustering across runs with different random seeds. Using a single A100 GPU we could generate one 144 megapixel (9k x 16k) panorama image in a few hours. VISTA interfaces for twelve Gemma Scope latents were created and can be explored in our online interface; three of these are explored below as case studies.We examined three specific latents to demonstrate the capabilities of our VISTA approach: Gemma Scope label: "ingredients and dishes related to food preparation and recipes" (link)Our findings: Visual inspection confirmed a large number of food-related ingredients, supporting the Gemma Scope label. The visualization hinted at specific ingredients triggering this latent code and their proportions in the dataset. For example, Figure 2 includes inputs referencing champagne, beer, iced tea, chips, honey, pizza, pretzels, and cheese. This map has a mutual-knn gain max of 0.129 at k=9%.Case Study 2: "muscle" [gemma-2-2b/20-res-16k/9745] Gemma Scope label: "references to muscle-related subjects and terminology" (link)Our findings (see Figure 1 ): Initial inspection agreed with the label, showing many regions with muscular imagery. However, we discovered an additional pattern: about 40% of the inputs lacked muscular references, instead clustering around words beginning with "M" (e.g., "musical", "mustache", "mystic", "mossy"). This suggests the latent is also weakly triggered by certain "M" words. This map has a stronger mutual-knn gain max of 0.273 at k=5%. Gemma Scope label: "expressions related to legal or financial obligations" (link)Our findings: This case was the most divergent from the Gemma Scope label. VISTA visualization showed clear visual clusters with no references to finance. Further examination revealed that this latent is triggered by conjunctions of visual forms, such as "a dramatic sunset or sunrise", "in a medieval or fantasy setting", "in a black & red attire", and "a unique half-deer, half-human figure". This discrepancy could be due to domain skew between datasets or limitations in automated techniques for identifying complex associative patterns. This map has a relatively weaker mutual-knn gain max of 0.099 at k=12%.These case studies demonstrate VISTA's ability to confirm automated interpretations and uncover additional, sometimes unexpected, patterns in SAE latents. 
conclusion: 
 VISTA (Visualization of Internal States and Their Associations) introduces a novel approach to exploring and interpreting neural representations through visual cartography. Our experiment with sparse latents from the Gemma2-2B model demonstrates VISTA's potential to complement and extend current automated interpretability techniques.Key findings include:• Confirmation of some automated interpretations, validating VISTA's basic functionality.• Discovery of additional patterns not captured by automated methods, such as secondary triggers for certain latents. • Revelation of unexpected associations, highlighting potential limitations in current automated techniques.These results suggest that visual, interactive exploration can provide valuable insights into complex neural representations. VISTA offers a promising direction for future research in AI interpretability, potentially bridging the gap between automated analysis and human intuition.As we continue to develop and refine VISTA, we anticipate its application to a broader range of neural representations and its integration with existing interpretability pipelines. This work represents an early step towards more comprehensive and intuitive understanding of neural networks, contributing to the broader goal of unifying representations across different models. 
 
 Appendix: We have attempted to quantitatively measure the fidelity of the resulting VISTA visualizations using mutual-knn scoring as is covered in more detail in (Huh et al., 2024) . We have found two conventions useful in adapting this metric:1) We generally express k as a percentage instead of an absolute number, which is easier to compare across different size datasets. We have also found in practice that mutual-knn can be quickly estimated my taking a smaller subset of the data, which is useful for iterative development. Each of our case study datasets were 4000 points, so absolute k reported can be found by multiplying (eg: mutual-knn at 1% refers here to k=40)2) We generally allow higher values of k (5-10%) than usually used found in alignment studies as we expect mapping down to two dimensions to be quite lossy. However for unaligned and randomly distributed data, the mutual-knn will grow linearly with k. So we report mutual knn "gain" -which subtracts off this k percentage expected by chance. Note that when k is expressed as a percentage, the maximum gain is now 1-k and misaligned datasets can even have negative gain up to -k. With this calibration, randomly distributed data has an expected mutual-knn gain of zero across all values of k.We've found these conventions useful in understanding how the alignment is behaving at various resolutions. For example, we can vary k to verify the mutual knn gain in case study one is maximized at k=9% and also characterize the degradation at finer and courser settings. (Figure 4 )Figure 4 : VISTA map mutual-knn gain in case study one with a maximum at k=9% (360)Below are higher quality versions of the three VISTA maps covered in case studies. These VISTA maps are also available online at http://got.drib.net/latents/ along with several others generated as part of this initial study. http://got.drib.net/latents/ 38th Conference on Neural Information Processing Systems (NeurIPS2024). 


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: Interpretability throguh 2D cartographic visualisation
main_review: main_review: The paper introduces a new framework (VISTA - Visualization of Internal States and Their Associations) for visualizing  (and potentially interpreting) neural representations of neural networks through what is referred to a "visual cartography" approach, i.e. the creation of a visual map that spatially represents the relationships between different elements of high-dimensional data.

The proposed pipeline goes as follows: 
After projecting the high-dimensional representations into a 2D space using UMAP, VISTA identifies clusters of similar representations. These clusters represent regions in the 2D map where data points with similar latent features are grouped together. Then it uses **text-to-image diffusion models** to generate visual representations (images) for each of these clusters as part of a MultiDiffusion panorama. This is supposed to confer interpretability since data points with similar semantic meaning are going to be visualised nearby in the 2D space as images of similar content. The resulting 2D map is designed to be interactive, allowing users to zoom in and explore or probe specific regions.



They apply the proposed approach on sparse latents obtained from the Gemma2-2B sparse autoencoder, where current state of the art techniques for interpretability rely on large language models. Instead the proposed approach proposes to probe to neural representations of autoencoder -like models in the visual domain instead of the language domain


**Strengths:**
- The method provides visual insights into neural representations.
- The method can be applied across various neural models and datasets, making it adaptable to different research contexts.
-  The "cartographic" nature of the approach allows for the creation of interactive visual maps that make it easier to intuitively explore the structure of the data.


**Weaknesses:**
- The authors do not provide any discussion on the limitation of the method.
- I find that the underlying challenge of aligning visual results with semantic meaning is not convincingly addressed. 
- Moreover I consider that the significance of the paper is a bit limited. While it addresses the important problem of interpreting high-dimensional neural representations, the practical impact of VISTA is unclear. The results presented in the paper, such as the identification of secondary triggers for certain latents, are interesting, but it is uncertain whether these insights could be reliably used to improve model interpretability. Furthermore, the reliance on visual inspection as a primary validation method makes it difficult to generalize the findings. VISTA might be of interest to researchers focused on interactive visualization, but as I understand it, it is unlikely to advance the state of the art in a substantial way, but I am happy to get convinced by the authors for the reverse.
- Also see questions.


**Questions:**

- The authors mention that they apply "a custom metric combining cosine distance in SAE space with the absolute difference along the latent axis". Can they provide a justification for this choice of metric?

- How do the authors ensure that the semantic labels and patterns identified by VISTA are not biased by the dataset or the text-to-image model used? Could they probably provide more details on the validation process?

- One question that I had while reading the paper, and I think it is the main point that needs clarification in order to make the acceptance of the paper reasonable, is what is the main insight that this method provides? As I understand similar contents are arranged nearby on the the 2D visualisation, as hopefully expected, but even ignoring the fact that the UMAP algorithm introduces some sort of randomness in the global arrangement of the clusters, I am not sure what sort of interpretability this method confers. 

- The authors mention that to understand representational alignment they employ a mutual nearest neighbor metric, that however they do not describe further, so it is not clear to me what exactly this is.

- How would the method perform in cases where the latents do not represent objects but more abstract notions and concepts that are difficult to visualize?

- In the third example how do the authors conclude that the label of the Gemma score is incorrect and their visualization is correct? How can they ensure the reliability of VISTA's interpretations—whether the visual clusters identified actually reflect meaningful structure or if they, are influenced by biases in the text-to-image generation process. I am willing to believe them, but `i am not sure how one conclude which method is providing correct interpretation over the other.

- Considering that UMAP is used to generate the 2D projection into the collage-like representation, I would expect some systematic comparison on **i)** how different runs of UMAP with same hyperparameters influence resulting maps, and **ii)** how different values of the hyperparameters influence the resulting visualisations.
title: title: Introduces a novel pipeline to visually explore LLMs' internal representation but lacks clarity
main_review: main_review: **Summary:** The paper introduces "VISTA", a novel pipeline to generate cartography of internal representation of LLMs for visual exploration and interpretation.

**Quality, clarity and originality:** The suggested pipeline is creative but it could’ve been great if the authors elaborated more on explaining the method of VISTA.

**Significance of the work:** The current best practice for understanding the meaning encoded in each latent is "automated interpretability", which provides suggested labels for each latent as it is entered into LLM. This work offers a unique approach to exploring the meaning of encoded latent visually.

**Strengths:**
- This paper touches on lots of topics that interest the workshop audience
- The paper introduces an interesting method to visually explore the internal representation of neural network models

**Weaknesses:**
- The paper is missing a detailed explanation of how to render cartographic renderings, which is the most important part of this work.
- This work lacks of ability to convince the reader why the approach that VISTA used is a better way to explore the internal representation.
title: title: Official Review of Submission30 by Reviewer hHaa
main_review: main_review: **Summary**

This paper introduces VISTA, a novel latent space visualization technique that provides a mosaic of summarizing clusters of input labels in a lower-dimensional representation. The authors apply this method to analyze the latent codes of sparse autoencoders, demonstrating its effectiveness in uncovering deeper meanings and associations that traditional automated techniques may overlook. VISTA aggregates labels of local neighborhoods in a smooth manner, facilitating the understanding of clusters containing multiple semantic meanings.

**Strengths and Weaknesses**

**Strengths**

1.  VISTA offers a significant improvement over standard latent space visualization techniques. While traditional methods often represent embeddings as point clouds labeled with input signals (e.g., Liu, Yang, et al., 2019), VISTA aggregates labels of local neighborhoods smoothly, enhancing the understanding of complex, multi-semantic clusters.

2. The method respects the uncertainty of latent space areas with no features by masking those regions. This approach preserves the topological properties of the latent manifolds learned by UMAP, ensuring a more accurate representation of the data structure.

**Weaknesses**

1. The "cartographic rendering" step, which subdivides the 2D representation of the latent space, may face challenges depending on the window size:
   - A large window size might over-generalize dense clusters, potentially obscuring semantic diversity.
   - A small window size could lead to excessive computational costs.
   
   A more thorough analysis of the tiling approach is needed to address these potential issues and optimize the balance between detail and efficiency.

2.  The reliance on generative models like MultiDiffusion introduces potential issues such as hallucinations or non-informative conditioning. These challenges could impact the data analysis capabilities of the method and should be addressed more explicitly.

3. While the paper focuses on text-based datasets, it lacks a comprehensive discussion on how VISTA might perform with other modalities such as images or audio, limiting its perceived versatility.

**Questions**

1. How do you anticipate VISTA will perform when applied to datasets in other modalities, such as images or audio? What modifications, if any, would be necessary to adapt the method for these different data types?

2. Could you provide more details on the "custom metric" used for UMAP? How was this metric developed, and what specific advantages does it offer over standard metrics in the context of your visualization technique?

3. Have you conducted any experiments or analysis on the optimal window size for the "cartographic rendering" step? How does the choice of window size affect the balance between computational efficiency and the preservation of semantic diversity in dense clusters?

**Limitations**

While the authors have briefly addressed the implications of their work, a more comprehensive analysis of the limitations would strengthen the paper. This could include:

1. An exploration of potential biases introduced by the visualization technique and how they might be mitigated.
2. An analysis of the computational resources required for VISTA compared to traditional visualization methods.
3. A consideration of how the method performs on datasets with highly imbalanced or sparse distributions in the latent space, which could affect the assumptions of UMAP.

Addressing these points would provide a more balanced view of VISTA's capabilities and potential areas for future improvement.

**Ethical concerns:** No

**Soundness:** 3 good

**Presentation:** 2 fair

**Contribution:** 3 fair