Measuring Human-CLIP Alignment at Different Abstraction Levels

Authors: pablo  hernández-cámara
jorge  vila-tomás
jes  ús malo
valero  laparra
Venue: ICLR_cc_2024_Workshop_Re-Align
Published: 2024-02-05 15:04:04

Abstract: Measuring the human alignment of trained models is gaining traction because it is not clear to which extent artificial image representations are proper models of the visual brain. Employing the CLIP model and some of its variants as a case study, we showcase the importance of using different abstraction levels in the experiments, because when measuring image distances, the differences between them can have lower or higher abstraction. This allows us to extract richer conclusions about the models while showing some interesting phenomena arising when analyzing the models in a depth-wise fashion. The conclusions extracted from our analysis identify the size of the patches in which the image is divided as the most important factor in achieving a high human alignment for all the abstraction levels. We found that the method used to compute the model distances is crucial to avoid alignment drops. Moreover, replacing the usual softmax activation with a sigmoid also increases the human alignment at all the abstractions especially in the last model layers. Surprisingly, training the model with Chinese captions or medical data gives more human-aligned models but only on low abstraction levels.

Main paper: 
 introduction: 
 Measuring the alignment of the human brain and artificial networks can lead to models with better generalization abilities, more robust to adversarial attacks, better cross-domain learning (Nanda et al., 2021; Aho et al., 2022; Fel et al., 2022; Sucholutsky & Griffiths, 2023; Muttenthaler et al., 2023) , and it can help neuroscientists to study the underlying principles of human behaviour (Brette, 2019; Martinez-Garcia et al., 2019; Allen et al., 2022; Conwell et al., 2022; Caucheteux & King, 2022; Hernández-Cámara et al., 2023) . The alignment with human behaviour can be analyzed at different complexities or abstraction levels. For example, when measuring the alignment with the way humans measure the distance between images, the distances can have very different scales because of the complexity of the images and/or the distortions. It is not the same measuring the distances between an image and the same image with a small quantity of noise than between images from different classes as shown in Figure 1 . Measuring alignment at only one level of abstraction and only in the model deepest layer may miss interesting behaviours that occur at other abstraction levels and/or in other model layers.Figure 1 : Example of the different abstractions when measuring image distances. Low abstraction: measuring distances between pairs of images where one is the reference and the other is a distorted version with a small noise (TID2013). Middle abstraction: triplet preferences between images of the same class but with semantic differences, such as the number of objects (NIGHTS perceptual). High abstraction: triplet preferences between objects of different classes (odd-one-out THINGS). Although there have been some works studying the human alignment of multimodal models, especially CLIP (Geirhos et al., 2021; Muttenthaler et al., 2022; 2023) , here we propose to analyze in detail the way some training procedures and model design details affect its human alignment. Interestingly, we do not only measure the alignment in a single abstraction level as the majority of works do, but we measure at different abstraction levels: from measuring distances between images with small noise distortions to intra-class distances between images with different semantic meanings and finishing with inter-class image distances. Moreover, we also analyze how the alignment changes depending on the layer depth and training/architecture factors. 
methods: 
 Following Sucholutsky et al. (2023) , we measure the alignment between the human brain and different neural networks, using behavioural data about image distances. We consider not only a single dataset but three different data sources of different abstraction levels. We measure how human alignment at these three abstraction levels changes layer by layer depending on different model design choices or training procedures. To do that, we successively isolate the different factors, fixing all but one, as shown in Table 1 . Each row describes one experiment, and each column lists the explored options. Analyzing all the possible combinations is not possible because they are not available.We use behavioural data about image similarity consisting of human-assessed distances between pairs of images, or human preference in the triplets scenario. We use data from three different abstraction levels of distortion, where the differences between the original image and the distorted versions span from lower to higher abstraction. We call the different levels: low, middle, and high. Figure 1 shows an example from each of the abstraction levels considered.For the low abstraction, we use a perceptual quality database, TID-2013 (Ponomarenko et al., 2015) . It consists of pairs of images where one is a reference image and the other is a noisy version of it, and the mean opinion score regarding how much humans see the difference between the two images. In the middle abstraction, we use the NIGHTS perceptual data (Fu et al., 2023) . It consists of intra-class synthetic image triplets where the differences between the images cover a wide range of variations that include semantic differences, such as object layouts, poses or quantities. We use the THINGS similarity dataset (Hebart et al., 2020) for the high abstraction. It is a subset of the whole THINGS dataset, comprising 4.70 million triplet odd-one-out similarity judgments for more than 1800 classes. It consists of triplets of inter-class images and therefore the differences are much more abstract. Appendix A shows more details and examples of the different datasets.We restrict ourselves to models trained by third-party institutions to avoid dependences on training procedures. We focus on evaluating CLIP Radford et al. (2021) model, trained to predict correspondences between images and their captions, and analyze how some model design variations or training procedures affect its human alignment at the three different abstractions. For the different model architectures, we analyze the base-patch16, base-patch32, large-patch14, and large-patch14-336 versions. We compare the effect of the final activation function comparing CLIP and SigLIP (Zhai et al., 2023) , which replaces the usual softmax activation function with a sigmoid. We analyze if the language used in the training has some effect on the alignment by comparing CLIP (trained in English) with Chinese-CLIP (Yang et al., 2022) (trained in Chinese). Finally, we analyze the effect of the training data by comparing CLIP (trained with natural images) with BiomedCLIP (Zhang et al., 2023) (trained with medical images and texts). Appendix B shows more details of the models.As a baseline, we used some low abstraction image quality assessment (IQA) models: the classical SSIM (Wang et al., 2004) , and more recent models based on neural networks such as LPIPS (Zhang et al., 2018) and PerceptNet (Hepburn et al., 2020) , which has biological inspiration. 
experiments and results: 
 Following the methodology stated in table 1 , we fixed all the factors but one in each experiment.We represent models that appear in more than one experiment always with the same color for easier comparison. All the measures have been done in a depth-wise fashion to facilitate the analysis in a more granular way, so all the figures show the model's depth at the x-axis. As some models have a different number of layers, to plot them all together the x-axis represents the percentage of the network (0% being the image space and 100% being the deepest layer -projection). We compute distances between images using the Euclidean metric normalizing the differences so that all features have unit mean (see appendix C for more details about the distance measurement procedure and its crucial relevance). Regarding the alignment measurement, the y-axis shows Spearman correlation between model distances and mean opinion score in the low-abstract level and accuracy between model distance-based preference and human triplet preference in the mid/high-abstract levels.3.1 DIFFERENT ARCHITECTURE First, we analyze how the alignment changes depending on the CLIP model architecture: number of layers (base vs large), patch size (base-patch16 vs base-patch32) and number of patches (large-patch14 vs large-patch14-336). Figure 2 shows that the base-patch32 variant gets the best result in the three abstraction levels but especially in the low abstract data. This shows that the patch size is the most important factor to consider to attain high human alignment.Although the differences between the other three model sizes are small, the large model with more patches large-patch14-336 achieves worse results across the three abstraction levels. Also, the more aligned model, base-patch32, shows the maximum human alignment around the 70% of the model depth for the mid and high abstract datasets showing an alignment decrease in the final layers before an increase in the projection. It shows a much more flat relation with layer depth in the TID2013 dataset before a final drop in the projection. Note that all the models consistently outperform all IQA baselines in the high abstract data and their middle layers also in the middle abstract. They are expected to perform worse than PerceptNet on TID-2013 because it was trained specifically on that abstraction level.Second, we analyze how the alignment changes depending on the CLIP last activation function.Figure 3 shows that the SigLIP model gets higher human alignment than CLIP in the low-abstract TID2013 although their projections get closer. In the mid-abstraction level, the CLIP model is more aligned in the early layers but less in the deeper layers. In the high-abstract data, both models perform similarly until the deepest layers, where the CLIP model surpasses the SigLIP. Moreover, it is interesting to notice that the SigLIP model shows a decrease of alignment in its first layer (embeddings) especially in the middle-level data but it does not happen in the low-level data, where it shows a big alignment increase. Both models outperform IQA baselines in the three abstract data, except for PerceptNet in the low-abstract scenario.Third, we analyze how the alignment changes depending on the text caption language. Surprisingly, Figure 4 shows that there are differences in the models when checked at the lowest abstraction, especially at early layers where CLIP with Chinese captions results in a more human-aligned model. However, these differences completely disappear when the models are tested at mid or high-abstraction levels. As before, it is interesting to note that both models outperform the SSIM and LPIPS across the three abstraction levels especially when measuring at their intermediate/final layers but not at the beginning of the model in the mid-abstract level.Figure 4 : Human alignment with TID2013 (left), NIGHTS (center), and THINGS (right) analyzed layer-by-layer depending on the language used in the training.Finally, we analyze how human alignment changes depending on the training data type. In Figure 5 we found that the CLIP model trained with medical data (med-CLIP) is much more human-aligned than the CLIP trained with natural data (nat-CLIP) in the low abstraction level scenario. This difference in alignment is reduced when tested in more abstract data. However, in the mid and high abstractions, nat-CLIP is more aligned than med-CLIP in the early layers, but in the final layers, med-CLIP again surpasses the model trained with natural data. When comparing against the baselines, med-CLIP performs better than both SSIM and LPIPS at the lower and higher levels, but both models are closely matched to LPIPS at the mid and high abstraction levels.  
discussion & conclussion: 
 Here we hypothesize some possible causes for the results described above. First, we found that the relative size of the image and the patches used in its analysis is critical for the alignment between humans and machines. For fixed-size images, bigger patches are better (blue curve is the best in Fig. 2 ), while for fixed-size patches, smaller images are better (green curve is better than purple curve in Fig. 2 ). Both effects are consistent: when the scene is partitioned in too-small regions the machine-human alignment is worse. We hypothesize there are optimal scales for scene analysis (see Appendix D Lowe (2004) ; Lazebnik et al. (2006) ): when using the wrong scale spatial relations between regions are harder to describe and hence the behavior of the model maybe less human.Another interesting result appears when no-normalization of the features over the datasets is considered before distance computation (Fig. 9 in Appendix C): there is a systematic alignment drop at mid-depth in all the models. This may make qualitative sense: the meaning of the visual features along the architecture should be close to low-level visual primitives in early layers, and be more abstract (closer to a conceptual description) in late layers where training considers the distance with the textual representation. Given this evolution, distances at different layers would be aligned with human (visual) opinion early in depth, but the relation may be obscured later in the architecture leading to a drop in the alignment. Preliminary inspection of the responses suggests that this behavior comes from the fact that a single feature takes most of the energy of the signal from around 50% depth. That singularity was the reason to propose difference normalization over the datasets in our experiments. This normalization-over-images has two interesting consequences: (1) the drop in alignment disappears, and, (2) the increase in alignment happens in later layers for higher abstractions for almost all the models. As shown in Fig 2 , for low-abstraction (left) a substantial increase in alignment happens as early as at 10% depth. However, for middle abstraction (center), the substantial increase happens about 30% depth, and for high abstraction (right) alignment remains small up to 50% depth. This is consistent with previously observed behaviour going from low-level to higher-level problems both in CNN and tranformers (Zeiler & Fergus, 2014; Ghiasi et al., 2022) .Regarding the activation function, a hypothesis for SigLIP's superior alignment over base CLIP in Fig. 3 is that the CLIP softmax enforces high output values to highlight the correct answer. Whereas, in the SigLIP sigmoid high values would saturate the non-linearity, thus the model will enforce outputs in the sigmoid range. Having a wide output range is problematic with out-of-distribution data due to a more unpredictable output scale. Additionally, SigLIP's authors state that the sigmoid stabilizes the training (Zhai et al., 2023) , which could lead to a better performance model, making it harder to directly relate the difference only to the different activation functions.The differences in human alignment between the CLIP models trained in different languages could be the result of language differences, however it could also be due to the different datasets, training procedures, pretraining, or tokenizers between the models. As opposed to the experiments on region sizes where we compare models with uniform training procedures from the same organization (OpenAI), this experiment uses models from different organizations. Therefore it is difficult to make conclusions on the effect of the language. That is a clear limitation of this experiment.Finally, regarding the nature of the training data, we found more differences in the low abstraction case. We hypothesize that medical images contain finer details (high-frequency) than natural images. Thus a model trained on medical images is more aligned with TID2013 where the distortions are different types of small-amplitude noises. Many medical models have historically used high-frequency noise textures to classify images (Castellano et al., 2004; Jaén-Lorites et al., 2022) .In conclusion, analyzing CLIP's human alignment for different abstraction level problems and at different model depths we found (1) a global drop in machine-human similarity for higher abstraction problems, and (2) after response normalization, the layer with biggest similarity shifts towards higher depths for higher abstraction levels. This highlights the relevance of a depth-wise analysis, suggest that deeper layers indeed correspond to higher abstraction, but also reveal the progressive departure between human and CLIP representations. (3) We found that for all the abstractions, the right balance between image and patch size is much more important than having more layers or more patches. Finally, we found that training the model with medical images instead of natural images leads to a much more human-aligned model in the low abstract scenario but, as happened with the language, it does not have clear effect at higher abstractions. Moreover, we showed that many of the analyzed models outperform state-of-the-art image quality models, such as SSIM or LPIPS.  
 
 Appendix: As stated in the text, we restricted ourselves to open models trained by third-party institutions to avoid dependence on training procedures. Therefore, not all the model factor combinations are available for its study. We focus on analyzing CLIP model (Radford et al., 2021) and different training procedures and architecture design variations. Table 2 shows the names and specific versions of the different analyzed models. We took all the models from Hugging Face Model Zoo1 .To compare different CLIP architectures we used four versions: base-16, base-32, large-14 and large-14-336. The base/large indicates if the model has 13 or 25 layers. The number after base/large indicates the pixel size of the patches in which the image is divided when used as input to the model. Therefore, base-16 indicates that the image is divided into 16 × 16 pixel patches and that the model has 13 layers. In general, all the models resize the images to 224 × 224 pixels before dividing it into patches except for the last model large-14-336, which resize the image to 336 × 336 pixels and therefore has more patches than large-14. To compare the different final activation functions, we used CLIP and SigLIP models. The only difference between them is that the SigLIP model replaces its final softmax activation function with a sigmoid function. To compare the training language we compare between CLIP and Chinese-CLIP, trained with Chinese captions instead of English ones. Finally, to compare the type of data we compare CLIP and BiomedCLIP, trained on public medial data instead of natural images. For these three experiments, we used the base-16 versions, so that they have 13 layers, resize the images to 224 × 224 pixels and divide them into 16 × 16 pixel patches.As stated in the text, we used a normalized Euclidean metric to measure distances between image model outputs: we normalize the difference in each feature so that all features have unit mean over a wide class of scenes. However, that is not the only option to compute the distances. Figure 9 shows how alignment depends on the different explored options to compute distances/similarities between images for the CLIP-base-16. More particularly, we compare between three distance measurements:• Euclidean: ||a -b|| • Normalized Euclidean: ||norm(a -b)|| • Cosine distance: 1 -a•b ||a||•||b||Naive use of plain Euclidean distance leads to a drop in the alignment for 50% depth of the models. We obtained that measuring Euclidean with the normalized differences is needed to avoid an alignment drop that if not happens at the middle layers. We found that this drop happens in all the CLIP model sizes except for the base-32, the one with the biggest patch size. We did not find differences between using the Euclidean distances or the usual cosine similarity (inner product between vectorized outputs over the norm of the outputs). These effects may come from: (i) the evolution of the meaning of the features (Fig. 10 .A) from low-level visual primitives in early layers to (less-visual) descriptions more suitable to be compared with textual representations in later layers. When the representation starts to be less-visual it is not surprising that plain Euclidean distance fails to reproduce human behavior. (ii) the balance between the spatial struture of the image and the proper scale to analyze it (Fig. 10 .B). Classical hand-crafted methods to extract image descriptors for successful classification used to operate at different scales, and there is a scale-hierarchy that depends on the spatial structure of the image Lowe (2004); Fei-Fei & Perona (2005) ; Lazebnik et al. (2006) . For instance, faces such as the one in Fig. 10 .B have certain objects of certain scale (eyes, nose, mouth) in specific spatial relations. At high resolution (analysis regions of small size) objects are hard to identify and spatial relations are hard to establish. This effect may be happening to the transformers in CLIP too. This may explain the critical effect of the patch size in relation to image size. https://huggingface.co/models 



Reviews: review: title: title: Measuring abstraction levels is interesting
main_review: main_review: ### Strengths
- Comparing human alignment between models across levels of abstraction and across layers is compelling and of interest to the community.
- The authors construct reasonable experiments that are potentially a good fit for a workshop.
- The tentative conclusions are prima facie plausible.


### Weaknesses
- The differences in human alignment between the CLIP models trained in different languages could be the result of language differences, however it could also be due to the different datasets, training procedures, pretraining, or tokenizers between the models. As it stands, these are not controlled for experimentally and it is therefore difficult to make conclusions.
- Similarly, the differences in accuracy between CLIP and SigCLIP (and medical images vs CLIP) make it hard to say whether the human alignment differences come from the activation function or the training recipe.
- Should we be worried about confounding inter vs intra class effects? The middle abstraction level consists of intra-class images only, the higher level of abstraction inter-class. 
- The LPIPS and SSIM baselines are the same across the TID2013 and NIGHTS datasets. This is probably a mistake? 

### Misc
- Fig. 1 has labeled low/middle/high complexity. Should this be low/middle/high abstraction?
- The CLIP b-16 and l-* models experience sharp decreases in alignment correlation at similar middle layers for the low-abstraction dataset, and then increase. Is this due to a resizing of the spatial activations?
strength_weakness: strength_weakness: See above.
title: title: Interesting new evaluation method and results, minor clarity issue
main_review: main_review: Novelty/interest to the community:
I believe that the idea of systematically evaluating models at different levels of abstraction will be of great interest to the realign workshop community. The approach taken by the authors is novel and they have an interesting finding that larger patch sizes have greater human alignment at all abstraction levels. 

Clarity:
Overall the paper was written clearly and the motivation, experiments, and conclusions were easy to follow. I did have trouble understanding exactly how the authors were getting outputs from their clip model that they could then compute correlation scores on. For example, the task for the first level of abstraction involves predicting the perceptual similarity of two images, but the authors did not explicitly state how they computed perceptual similarity using clip embeddings for each image. I assume they are computing cosine similarity, but I recommend the authors include the full procedure for making a judgement using clip for each level of abstraction task in the appendix.

Correctness:
The experiments and analysis appear to be correct.
strength_weakness: strength_weakness: The work is well motivated. The evaluation approach and results will be of interest to the community.
title: title: A paper that explores alignment in CLIP models from the perspective of image similarity with a substantial number of experiments on a limited range of models.
main_review: main_review: Strengths: 
- Framing the question of alignment in terms of model and human assessment of image distance is interesting and allows for nice quantitative experiments. 
- Given that this is only a 5 page paper, a substantial number of experiments are run, allowing a reasonably thorough exploration.

Weaknesses:
- One of the challenges of studying foundation models such as CLIP is that it is very hard to perform experiments that control for training details. For instance, do the results of an experiment on two CLIP models (independently trained) arise from the training/architecture differences we are aware of or do they arise from nuances of training/data that are not publicly known. Further, we rarely have multiple instances of a model with fixed characteristics, so it is hard to determine what is training noise. This is mostly unavoidable given the compute budgets of most research groups, but it should definitely be cited as a limitation and conclusions should be framed accordingly.
- There are quite a few details that should be spelled out more completely for clarity. For instance, not all readers will be familiar with the different versions of the CLIP architecture. This should be spelled out explicitly, especially since these architectural features are central to some of the experiments. The details around the three datasets that are used for evaluation should also be stated since not all readers will be familiar with these. More example images in the appendix would be useful.

Nitpicks: 
- When describing the TID-2013 dataset, it might be useful to say a little more what the ‘opinion score’ is (e.g., what kinds of values does it take, is it continuous or discrete?)
- There is a space missing in the last paragraph of Section 2.1, “… images.In the…”

Questions: 
- What is a model’s “inner domain” (first paragraph, page 1)?
- What is the difference between large-patch14 and large-partch14-336?
strength_weakness: strength_weakness: The research direction is interesting and many of the initial findings are a strong foundation for future work.

Comments: comment: > One of the challenges of studying foundation models such as CLIP is that it is very hard to perform experiments that control for training details. For instance, do the results of an experiment on two CLIP models (independently trained) arise from the training/architecture differences we are aware of or do they arise from nuances of training/data that are not publicly known. Further, we rarely have multiple instances of a model with fixed characteristics, so it is hard to determine what is training noise. This is mostly unavoidable given the compute budgets of most research groups, but it should definitely be cited as a limitation and conclusions should be framed accordingly.

We agree with the statement and in the camera-ready version, we will cite it as a limitation. Although training our models will be the best option to be sure where the differences come from, it will create our own bias regarding the training procedure/data. Moreover, as the reviewer says, the computational power needed is prohibitive and therefore we decided to use the more similar models as possible. 

> There are quite a few details that should be spelled out more completely for clarity. For instance, not all readers will be familiar with the different versions of the CLIP architecture. This should be spelled out explicitly, especially since these architectural features are central to some of the experiments. The details around the three datasets that are used for evaluation should also be stated since not all readers will be familiar with these. More example images in the appendix would be useful. The details around the three datasets that are used for evaluation should also be stated since not all readers will be familiar with these. More example images in the appendix would be useful.

In the camera-ready version, we extended Appendix B Models to explain in detail the differences between all the analyzed models. We also added a new Appendix A Datasets with more examples and details about the different data used at each abstraction level.

> When describing the TID-2013 dataset, it might be useful to say a little more what the ‘opinion score’ is (e.g., what kinds of values does it take, is it continuous or discrete?)

We described it in more detail in the new appendix A Datasets.

> There is a space missing in the last paragraph of Section 2.1, “… images.In the…”

Thanks, changed.

> What is a model’s “inner domain” (first paragraph, page 1)?

We wanted to say the model’s inner representation, i.e. the outputs of the final/deepest model layer. To be more precise, we changed it to “model’s deepest layer”.

> What is the difference between large-patch14 and large-partch14-336?

The only difference is the number of patches: CLIP models usually resize images to 224x224, while the large-pathc14-336 resizes it to 336x336. Both models have a patch size of 14 so the only difference is the overall number of patches.
> Overall the paper was written clearly and the motivation, experiments, and conclusions were easy to follow. I did have trouble understanding exactly how the authors were getting outputs from their clip model that they could then compute correlation scores on. For example, the task for the first level of abstraction involves predicting the perceptual similarity of two images, but the authors did not explicitly state how they computed perceptual similarity using clip embeddings for each image. I assume they are computing cosine similarity, but I recommend the authors include the full procedure for making a judgement using clip for each level of abstraction task in the appendix.

For the camera-ready version, we have included information about how we measure the model distances and human alignment at each abstraction level at the beginning of section 3. Experiments. Also, we have included an appendix about different distance measurement procedures and how they affect the alignment.
> The differences in human alignment between the CLIP models trained in different languages could be the result of language differences, however it could also be due to the different datasets, training procedures, pretraining, or tokenizers between the models. As it stands, these are not controlled for experimentally and it is therefore difficult to make conclusions. Similarly, the differences in accuracy between CLIP and SigCLIP (and medical images vs CLIP) make it hard to say whether the human alignment differences come from the activation function or the training recipe.

We agree with the statement and in the camera-ready version, we will cite it as a limitation. Only the different size analysis uses models from the same organization (OpenAI) while the rest of the comparisons analyze models from different organizations and therefore follow different training receipts. Although training our models will be the best option to be sure where the differences come from, it will create our own bias regarding the training procedure/data. Moreover, as reviewer 3 says, the computational power needed is prohibitive and therefore we decided to use the more similar models as possible. 

> Should we be worried about confounding inter vs intra class effects? The middle abstraction level consists of intra-class images only, the higher level of abstraction inter-class.

We are not really sure about what the reviewer means with this question. On the one hand, the middle abstraction level consists of intra-class images where the differences include different image layouts, poses and number of objects. On the other hand, the high abstract level consists of inter-class image triplets, where each image is from a different class and therefore the differences between them are of higher level than in the middle abstraction level.

> The LPIPS and SSIM baselines are the same across the TID2013 and NIGHTS datasets. This is probably a mistake?

No, the values are correct. Notice that although they are similar values (but not the same), on TID2013 they represent correlations while on NIGHTS they are accuracies.

> Fig. 1 has labeled low/middle/high complexity. Should this be low/middle/high abstraction?

Yes, thanks, we have changed it to be more consistent in the camera-ready version.

> The CLIP b-16 and l-* models experience sharp decreases in alignment correlation at similar middle layers for the low-abstraction dataset, and then increase. Is this due to a resizing of the spatial activations?

No, there is no resizing of spatial activations in any model. We now know that the sharp alignment drop is due to the value explosion of one feature and that if we normalize the output so that all the features have the same mean this decrease disappears. The new appendix C Distance Mesarumenets explains it in more detail. We are still working on understanding why this feature suddenly increases its value from a given layer.