dataesr/scientific-paragraphs-categorization huggingface dataset:
Quantitative Evaluation Results We investigate the following research questions: 1. Does a controlled natural language as target for KGQA semantic parsing using LLMs have a positive effect on accuracy? 2. Are smaller amounts of training data sufficient to teach an LLM a controlled natural language as opposed to a formal language? 3. Does the answer to the above two questions depend on the choice of language model? 4. Does the answer to the first question depend on the question type? We will walk through our research questions and connect them to the quantitative evaluation results we obtained. The first research question is whether using CNL can result in improved accuracy. We report our results in Table  2. Mintaka is generally a challenging dataset with hits@1 scores reported  [25]  to be between 0.
12 and 0. 2 on its test set for 3 different approaches. (Note that the Mintaka test set is a superset of our Mintaka-CNL test set and the reported approaches use weak supervision rather than strong supervision. Therefore, the results are not directly comparable. ) The hits@1 results for the two controlled natural languages, Sparklis and SQUALL, are both higher than the SPARQL baseline. Using SQUALL leads to higher accuracies for all models except T5, which could be due to it being even closer to natural language than Sparklis and generally being more compact. For the top-performing model, the accuracy using SQUALL is roughly twice as high compared to using SPARQL, so the choice of target language has a strong influence on the performance of the KGQA system. For the string similarity proxy metrics (BLEU, METEOR, ROUGE) SQUALL and Sparklis are similar, and consistently higher than SPARQL.
Model Setting Language BLEU METEOR ROUGE Exact Match Hits@1 Δ Hits@1 GPT-  Sparklis is less expressive than SPARQL and SQUALL. This means that the ground truth queries may not capture all aspects of the input question (e. g. rather than selecting the "5th US president" the Sparklis query returns "US presidents" in increasing "point in time"). It can also be an underestimate since different query surface forms can lead to the same execution results. For both SPARQL and SQUALL, we observed that accuracy is usually much higher than the exact match score. We also observed that the LLM sometimes generates SQUALL queries that are substantially different from the ground truth, but are semantically equally good and often lead to the same execution results. If we take this ability to generate correct but syntactically different queries as a proxy for the LLM obtaining a deeper understand- query not matching KG structure, which uses a specific entity type <Star Trek films> ing of the target language (as opposed to simpler expression matching) then the pattern we see is: more natural target languages (see Figure  1 ) lead to a deeper understanding of the language by the large language model.
To answer the second research question, we varied the number of fine-tuning examples and plotted execution accuracies in Figure  3  for the two best performing LLMs. We observe that SQUALL is generally more accurate across all data sizes. For lower amounts of fine-tuning data SPARQL is more accurate than Sparklis for GPT-3 Davinci -probably because it was seen more in pre-training. Increasing the number of fine-tuning examples leads to Sparklis performing slightly better than SPARQL for GPT-3 Davinci with a constant margin. From our data, we cannot generally conclude that using a controlled natural language always reduces training data requirements -it is the case for SQUALL but to a lesser degree for Sparklis. So whether or not training data requirements for a CNL are substantially lower appears to be language-specific.
SQUALL is both more natural and more compact than Sparklis, i. e. it can express a query in less tokens, which could explain the performance gap. For the third research question, we computed the relative improvement for each language model in the execution accuracy of SQUALL compared to SPARQL in the last column of Table  2. It is interesting and surprising that the improvement from a formal language to a controlled natural language does not seem to depend much on the choice of the language model. Even for a small model like the GPT-2 124 million parameter model, which could not generate a single accurate SPARQL query, we observed substantial absolute improvements when using a controlled natural language. For the fourth research question, we analyzed the different types of questions in Mintaka in terms of absolute performance and performance improvement when using SQUALL instead of SPARQL.
Conclusions and Future Work We verified our research hypothesis that using a controlled natural language as target for LLM-based KGQA semantic parsing provides advantages compared to translating into a logical form directly. This appears to hold despite the logical form being more prevalent in use and therefore in pre-training data. Our study with two different CNL shows that the more compact and natural SQUALL language leads to much better results compared to a more regular, verbose Sparklis language. Using the most powerful LLM in our test, the semantic parsing accuracy could be doubled by switching from SPARQL to SQUALL. We could also observe that the LLM likely obtains a better understanding of SQUALL, e. g. it can generate correct syntactic variations of the ground truth and very rarely generates meaningless or duplicated patterns.
Acknowledgements This work is partly supported by the  BMBF (Federal Ministry of Education and Research)  and  DAAD (German Academic Exchange Service)  in project  SECAI  (grant  57616814 ). It is also supported by  EU  projects  CALLISTO  ( EU  grant  101004152 ) and  e-Vita  ( EU grant   101016453 ) as well as the ScaDS. AI ( IS18026A-F ) center for providing High Performance Computing infrastructure access at  TU Dresden, Germany.
Recent work has argued that large-scale neural recordings are often well described by low-dimensional 'latent' dynamics identified using dimensionality reduction. However, the view that task-relevant variability is shared across neurons misses other types of structure underlying behavior, including stereotyped neural sequences or slowly evolving latent spaces. To address this, we introduce a new framework that simultaneously accounts for variability that is shared across neurons, trials, or time. To identify and demix these covariability classes, we develop a new unsupervised dimensionality reduction method for neural data tensors called sliceTCA. In three example datasets, including motor cortical dynamics during a classic reaching task and recent multi-region recordings from the International Brain Laboratory, we show that sliceTCA can capture more task-relevant structure in neural data using fewer components than traditional methods.
, 2016 , Pastalkova et al. , 2008 , Peters et al. , 2014 , Okubo et al. , 2015 , Harvey et al. , 2012]. Second, the neural encoding weights for a given sensory stimulus may change over trials due to adaptation, learning,  [Hennig et al. , 2021] , or representational drift  [Rule et al. , 2019 , Driscoll et al. , 2017 , Schoonover et al. , 2021]. Because methods such as PCA or TCA look for covariability across neurons, they may miss additional forms of variability that are instead shared across time or across trials. To address this, we introduce slice tensor component analysis (sliceTCA), an unsupervised dimensionality reduction method that is able to identify and disentangle latent variables belonging to three di↵erent classes 1 of covariability (defined as variability shared across neurons, time, or trials) that are mixed within the same dataset.
This property contrasts sliceTCA from matrix factorizations like PCA which capture a single covariability class at a time, and from TCA which identifies components constrained to all of them simultaneously. As a result, we show that sliceTCA can capture more structure in fewer components than either of these methods. Based on theoretical and practical considerations of the sliceTCA decomposition, we develop an analysis pipeline for model selection, optimization, and visualization that is implemented in a readily applicable Python library. After validating our method on simulated data, we illustrate the utility of sliceTCA in three large-scale neural datasets. First, we demonstrate that di↵erent covariability classes encode distinct behaviorally relevant neural dynamics in motor cortical recordings in non-human primates  [Churchland et al.
, 2012]. Next, in simultaneous imaging data from cortical and cerebellar populations during a cued motor task  [Wagner et al. , 2019] , we show that sliceTCA untangles task-relevant manifolds by taking into account covariability across trials. Finally, we analyze a recent dataset from the International Brain Laboratory  [IBL et al. , 2022]  and show that sliceTCA disentangles region-specific covariability classes across visual cortex, hippocampus, thalamus, and the midbrain. We then provide a geometric intuition for how neural population activity is shaped by latent variables belonging to the three di↵erent covariability classes. Together, these results demonstrate the necessity of extending the traditional view of latent variables and neural covariability to uncover higher-dimensional latent structure.
This could represent common dynamics whose neural encoding weights change from trial to trial, e. g. , due to learning, adaptation, or drift (Figure  1e ). If only one of these three slice types were fitted, sliceTCA would be equivalent to a matrix factorization on the respective unfolding of the data tensor (Supplementary Figure  1a-c ). Indeed, previous work has argued for performing PCA on di↵erent unfoldings of the data tensor to identify the slice type that gives the best approximation  [Seely et al. , 2016]. Crucially, sliceTCA di↵ers from this approach by fitting all three slice types simultaneously, thereby demixing di↵erent covariability classes that may be combined within the same dataset (Figure  1f ,g). SliceTCA is also related to, yet distinct from TCA (also known as CANDECOMP/PARAFAC or CP decomposition)  [Williams et al.
Task relevant information is distributed across slice types in motor cortical reaching dynamics Based on the results of our toy model, we predicted that di↵erent slice types could capture di↵erent kinds of behaviorally relevant dynamics in neural data. We tested this hypothesis in a dataset comprising primary motor cortical (M1) and premotor cortical (PMd) populations recorded simultaneously during maze reaching and classic center-out (no maze) reaching tasks (Figure  2a , hand position). To quantify decoding performance, we linearly mapped population activity onto hand velocity (Methods). As a benchmark, we first mapped trial-averaged raw neural data on kinematic trajectories, revealing a close match between behavior and neural activity (Figure  2a , trial-averaged raw data). However, when we attempted to decode hand trajectories based on individual trials, we observed significant trial-to-trial variability that corresponded poorly to kinematic data (Figure  2a , raw data).
Remarkably, this simple change in denoising strategy resulted in a significantly better match between trial-to-trial variability in the data and in the hand kinematics (Figure  2a , trial-slicing NMF). We further validated that the components obtained by trial-slicing NMF corresponded to reach-tuned sequences whose temporal orderings were reproducible across held-out data (Supplementary Figure  3 ). These results reveal that in this dataset, behaviorally relevant information was encoded in neural sequences shared over trials, rather than by shared variability across neurons. Trial-and neuron-concatenated NMF constitute two special cases of non-negative sliceTCA, where either neuron-slicing components or trial-slicing components exclusively are fitted. Therefore, we next asked whether we could identify additional information in the data by demixing di↵erent classes of covariability with sliceTCA.
We have further identified a second class of invariant transformations that is specific to sliceTCA (Supplementary Figure  7b , 8, Methods). This invariance class, when unaccounted for, prevents an unambiguous attribution of covariance patterns to one of the three component types. Such invariances can lead to di culties in comparing latent representations across multiple datasets  [Williams et al. , 2021]  and are therefore crucial to address for any new method. To address these concerns and to provide a user-friendly guideline for sliceTCA, we developed a full data analysis pipeline for sliceTCA including data preprocessing, model selection, model optimization, and visualization (Figure  3a ). First, trials must be time-warped, trimmed, or masked, in order for the data to be shaped into a tensor.
Denoising task-relevant manifolds during a motor task With a standardized data analysis pipeline established, we next asked how behaviorally relevant latent structure sliceTCA could uncover in a novel dataset, without any prior expectation on the component types. We applied sliceTCA to a dataset consisting of simultaneously imaged cerebellar granule cells and pyramidal neurons in the premotor cortex of mice performing a motor task (Figure  4a )  [Wagner et al. , 2019]. Using the sliceTCA analysis pipeline, we selected a model with three trial-slicing components and three neuron-slicing components at the elbow of the cross-validated loss function (Figure  4b ,c, Supplementary Figure  11 ; similar components observed in the optimal model, Supplementary Figure  12 ). The first trial-slicing component captured temporally distributed cerebellar and cortical dynamics that were common to both left and right correct reaches, but distinct from error reaches (Figure  4b, d ).
Identifying components with region-specific covariability patterns in multiregion recordings So far we have shown that mixed variability co-occurs within the same neural population. However, the need to consider multiple covariability classes becomes even more crucial in simultaneous recordings from many regions, as previous work has shown that di↵erent brain regions may be better described by di↵erent unfoldings of the data tensor  [Seely et al. , 2016 ]. Yet, relying on di↵erent tensor unfoldings for the analysis of distinct regions would require that these regions be analyzed separately, without leveraging the simultaneous nature of such data. We therefore asked whether sliceTCA could demix area-specific representations in distinct slice types. To test this idea, we took advantage of a recently published dataset consisting of Neuropixel recordings across six brain regions during a perceptual decision-making task (Figure  5a )  [IBL et al.
3 Discussion Neural population dynamics are frequently interpreted as low-dimensional latent variables encoded by fixed subgroups of neurons, which represent shared variability across neurons. Here, we have advocated for an expansion of this view of structure in neural data which takes into account three distinct classes of shared variability: across neurons, time, and trials. Towards this end, we introduced sliceTCA, a new tensor decomposition method that is able to demix latent variables that belong to any of these covariability classes. Through several example datasets, we demonstrated that sliceTCA can capture more task-relevant covariability in neural data in fewer components, enabling the description and interpretation of complex latent structure embedded in large-scale neural recordings.
, 2016]. Our results support this hypothesis, and further show that these di↵erent classes can be demixed by sliceTCA. Therefore, demixing covariability classes may be a crucial step when considering large-scale multi-region recordings  [IBL et al. , 2022 , Wagner et al. , 2019 , Ebrahimi et al. , 2022 , Ahrens et al. , 2012]. A longstanding challenge in systems neuroscience is the di culty of mapping neural variability to changes in behavior  [Renart and Machens, 2014]. This can be accomplished using supervised dimensionality reduction methods that use information regarding behavior or task outcomes to identify latent variables  [Kobak et al. , 2016 , Sani et al. , 2021a , Balzani et al. , 2022]. Despite it being an unsupervised method, we found that SliceTCA was able to disentangle behavioral and task information in each of the datasets presented.
We claim that this is due to two reasons: first, demixing di↵erent sources of covariability e↵ectively "denoises" components that represent task variables that would have otherwise been occluded by additional sources of variability. Second, the trial-slicing components explicitly identify dynamics that are shared across trials, which tend to be defined by task variables or behavioral outcomes. Indeed, in each of the three datasets, we found that trial-slicing and time-slicing components correlated with behavioral variables. Moreover, in our feedforward model, we suggest how sliceTCA could o↵er a window into the computational roles of variables modeled by di↵erent slice types. We argue that the classical view on neural latents, which assumes that a key part of behaviorally relevant neural variability is correlated across neurons, is overly reductionist and may miss many types of neural dynamics underlying behavior.
Identifying invariances is crucial for reproducibility and interpretation, as non-unique solutions may prohibit clear comparison across datasets  [Dyer et al. , 2017 , Gallego et al. , 2020]. This issue is ever more important with the recent increase in popularity of comparisons of neural data to task-trained neural network models, whose representations are known to be sensitive to model specifications such as architecture and inputs  [Lindsay et al. , 2022 , Williams et al. , 2021]. Going forward, matrix and tensor decompositions could prove useful for comparing latent representations by virtue of their interpretability and tractability. SliceTCA falls into a larger class of tensor decomposition methods including TCA  [Williams et al. , 2018 , Harshman et al. , 1970] , which captures variability lying at the intersection of the covariabilty classes, and the Tucker decomposition, which allows factors to interact via a core tensor  [Onken et al.
By construction, the tensor X has slice rank of 2, as it can be written in the following form: We fitted sliceTCA with non-negativity constraints to the synthetic dataset, using five blocks of 200 X = I S + I TD iterations each with a learning rate which decayed exponentially over blocks from 0. 2 to 0. 0125, and a mask that decayed exponentially over blocks from 0. 8 to 0. 05. Masked entries changed randomly every iteration. We analyzed a dataset of motor cortical (M1, N = 90) and premotor cortical electrophysiological recordings (PMd, N = 92)  [Churchland et al. , 2012] , which is curated and publicly available as part of the 'Neural Latents Benchmark' project  [Pei et al. , 2021]. Briefly, monkeys were trained to perform a delayed center-out reach task to one of 27 locations in both maze conditions (in which barriers were placed on the screen, leading to curved optimal reach trajectories) and in no maze conditions with matched target locations (classic center-out task leading to straight optimal reach trajectories).
Matrix rank of slices To determine whether sliceTCA finds components with higher matrix rank than methods that do not demix slice types (neuron-slicing PCA and factor analysis (FA) with neuron loadings, neuron-time-concatenated PCA and FA with trial loadings), we performed singular value decomposition (SVD) on the six slices of the sliceTCA model shown in Figure Figure  4b , as well as on the scores of either trial-slicing or neuronslicing PCA and FA, after refolding the resulting scores into N ⇥ T or K ⇥ T matrices, respectively. We then compare these to the normalized eigenvalue spectra of the slices of the trial-slicing (Figure  4e ) or neuron-slicing components (Supplementary Figure  13d ). Factor analysis was performed using the Python package "sklearn"  [Pedregosa et al. , 2011] , which uses an SVD-based solver.
Acknowledgements We thank  Joao Barbosa ,  Matthias Hennig ,  Kishore Kuchibhotla ,  Ashok Litwin-Kumar ,  Arno Onken ,  Yann Sweeney , and the members of the  Cayco Gajic lab  for comments on the manuscript. We additionally thank  Angus Chadwick  for helpful discussions on an early stage of the manuscript, and  Mark Wagner  for sharing his data. We are also grateful to the  IBL , the  Churchland/Shenoy labs , and the  Neural Latents Benchmark  project for making their processed and curated data freely available. This work was funded by  EMBO  (H. S. ,  ALTF 471-2021 ) and the  Agence Nationale de la Recherche  (N. A. C. G. ,  ANR-20-CE37-0004 ;  ANR-17-EURE-0017 ).
Xn,t,k      where Xneuron describes the partial reconstruction of the data tensor from only the neuron-slicing components. We similarly defined the contributions of the time-and trial-slicing components to the sliceTCA reconstruction of each neuron n as f time n Python library for the sliceTCA data analysis pipeline (including preprocessing, model selection, model optimization, and visualization of components) is available at https://github. com/arthurpe/slicetca. In addition, the code necessary for reproducing main analyses will be published in a separate Github repository upon publication.
The dashed line represents the mean squared deviation between the noisy and the noiseless model. Adding white noise does not a↵ect the performance of sliceTCA relative to TCA or PCA. of neuron-specific sequences identified in trial-slicing components. a Components identified by trial-slicing NMF (Rneuron = 6) applied to the motor cortical reaching dataset (Figure 2). To validate the sequences we separated the data into even (left) and odd (right) trials and fit the model separately on each of the two datasets (only 6 rather than 12 components to avoid overfitting since we halved the datasets). Components from the two models were matched by hand based on similarity of the trial loading vector. In each row of the panel, neurons in the slices are sorted identically (according to the latency of peak activity in even trials).
Babu et al. [21]  focused on the angular error and performance optimization as a function of the mirror width, and Zhu  [22]  developed an analytical optical approach, named FirstOPTICS, which allows for the evaluation of the optical performance of linear Fresnel collectors. On the other hand, Chaitanya et al. [23]  used a Monte Carlo ray-tracing method for the optimization of a solar linear Fresnel reflector and in particular the secondary reflector to achieve uniform flux distribution over the absorber tube. Balaji et al. [6]  applied the Monte Carlo ray-tracing method to evaluate the optical performance of a linear Fresnel prototype in India. In the case of the eLLO power plant, the optical performance of the module was characterized using the Monte Carlo ray-tracing method in  [4]. From a thermal perspective, the thermohydraulic behavior of these receivers was frequently characterized by the 3D modeling using the CFD software as well as 1D mathematical models.
Thus, a 1D single-phase thermohydraulic model based on MATLAB is developed to express theoretical heat loss correlations, however, limiting it to the influence of the wind speed for the sake of simplicity. Finally, an empirical correlation of the linear heat losses of the eLLO receiver is ex pressed in Equation (  4 ) as a second-degree polynomial according to the difference be tween the inlet/outlet fluid temperature average and the ambient temperature. This correlation is compared in the following section with the numerical correlation expressed using the 1D single-phase thermohydraulic model.