dataesr/scientific-paragraphs-categorization huggingface dataset:
Data modeling First, we note that throughout the paper, vectors are given in bold capital letters, sizes in capital letters, and indices in lowercase. To start with the missing data problem, a network dedicated to image time series is usually fed a sequence of images I ∈ R H×W×5 , where H and W are, respectively, the height and width of the image and 5 is the number of channels representing the bands (u, g, r, i, z). However, we know, as explained earlier, some bands are missing in the dataset. To fix this issue, instead of giving the model images with empty channels, thus introducing a bias to the network, we decided to separate the channels as individual images (X ∈ R H×W ) simply by skipping the empty channels. As a consequence, the information about the type of filter, which holds a crucial value for the network to accurately discriminate between objects, is also eliminated.
In an image with different channels, the order of the channels usually represents the type of filter (see Fig. 2 ). In order to preserve this valuable information, we should add the band type to the new 2D images X. Knowing that the information about the type of the filter is a categorical feature, thus we need to adapt it to the model 2D input representation. To do so, we propose using an embedding layer to encode the channel type before passing the input to the model. For each band (u, g, r, i, z), we assign a unique number id ∈ {1, 2, 3, 4, 5}. Then, an embedding layer BandEmbed converts the band type id, which is a categorical feature, into 2D dense representation E id with E id ∈ R H×W (see Fig. 4 ): E id = BandEmbed(id). (1) The embedding layer is a fully connected layer that is reshaped to a 2D representation.
However, this approach tends to produce a vanishing gradient during backpropagation  (Hossain et al. 2021). Other solutions propose the use of oversampling such as SMOTE  (Chawla et al. 2002). Those authors proposed an approach where they synthesize new samples of the minority class. However, this solution was proposed mainly for tabular data. Knowing that our data are images that contain a much higher number of features than tabular data, it appears obvious that using SMOTE may not be optimal in our case. Dablain et al. (2021)  introduced a solution based on SMOTE dedicated to images called DeepSMOTE. It is aimed at generating new images for the minority class. Once again, this approach is unsuitable in our case as our dataset is not composed of images, but of a sequence of images, and it is too expensive to generate a whole new sequence.
So, instead of generating a new one, we used data augmentation and weighted random sampling(WRS)  (Efraimidis 2015)  on our database. We oversampled the dataset, which translates to simply altering the dataset to remove such an imbalance by increasing the number of minority classes and undersampling the data by decreasing the majority classes until we have reached a balanced dataset. In our case, the WRS was applied on a batch level. We generate balanced batches based on the probability of a sample being selected. We weighted each sample according to the inverse frequency of its label's occurrence and then sampled mini-batches from a multinomial distribution based on these weights. This means that samples with high weights are sampled more often for each mini-batch. The same sample can be reused in other mini-batches of the same epoch to increase the minority class, but with a data augmentation applied to it.
Different methods of data augmentation were used: for example, a random drop of some steps from the whole sequence to create a new one or a sequence rotation, horizontal and vertical flips, and sequence shifting, where we construct a smaller sequence from the original one which has a bigger length than the input length of ConvEntion. In our implementation, we recall the dataset at every epoch, the transforms operation (augmentation) is executed and then we get different augmented data. Using this oversampling approach has drastically improved the performance of the model. We used the function WeightedRandomS ampler from PyTorch  (Paszke et al. 2019)  as an implementation of WRS.
Wang Article number, page 5 of 11 the cross-entropy loss in order to focus the learning on hard misclassified examples. However, this approach tends to produce a vanishing gradient during backpropagation  (Hossain et al. 2021). Other solutions propose the use of oversampling such as SMOTE  (Chawla et al. 2002). Those authors proposed an approach where they synthesize new samples of the minority class. However, this solution was proposed mainly for tabular data. Knowing that our data are images that contain a much higher number of features than tabular data, it appears obvious that using SMOTE may not be optimal in our case. Dablain et al. (2021)  introduced a solution based on SMOTE dedicated to images called DeepSMOTE. It is aimed at generating new images for the minority class. Once again, this approach is unsuitable in our case as our dataset is not composed of images, but of a sequence of images, and it is too expensive to generate a whole new sequence.
So, instead of generating a new one, we used data augmentation and weighted random sampling (WRS; Efraimidis 2010) on our database. We oversampled the dataset, which translates to simply altering the dataset to remove such an imbalance by increasing the number of minority classes and undersampling the data by decreasing the majority classes until we have reached a balanced dataset. In our case, the WRS was applied on a batch level. We generate balanced batches based on the probability of a sample being selected. We weighted each sample according to the inverse frequency of its label's occurrence and then sampled mini-batches from a multinomial distribution based on these weights. This means that samples with high weights are sampled more often for each mini-batch. The same sample can be reused in other minibatches of the same epoch to increase the minority class, but with a data augmentation applied to it.
Different methods of data augmentation were used: for example, a random drop of some steps from the whole sequence to create a new one or a sequence rotation, horizontal and vertical flips, and sequence shifting, where we construct a smaller sequence from the original one which has a bigger length than the input length of ConvEntion. In our implementation, we recall the dataset at every epoch, the transforms operation (augmentation) is executed and then we get different augmented data. Using this oversampling approach has drastically improved the performance of the model. We used the function WeightedRandomS ampler from PyTorch  (Paszke et al. 2019 ) as an implementation of WRS.
(2014) , where it applies a 3D filter to the dataset and the filter moves in three directions to calculate the low-level feature representations. Their output shape is in a 3D volume space. We applied 3DCNN where we input the sequence J to get the reduced new sequence S following the equation: S n = 3DCNN(J (n-1) * K+1 ,. , J n * K ), n ∈ {1,. , N}. (3) We let M be the length of the series, J and we fed K inputs of J to the 3DCNN to generate one entry, S , for our transformer. So, in the end, the new sequence, S , will be S ∈ R N×H ′ ×W ′ ×D , where N = M/K, D is the number of channels and H ′ and W ′ are the new height and width. By using the 3DCNN, we reduced the length of the sequence by a factor of K, which also reduced the complexity of the model. The 3DCNN does not just reduce the length of the input sequence, it also captures local spatio-temporal low-level features.
The sinusoidal positional encoding is chosen to make it easy for the model to learn to attend to relative positions. To get the new input for the convolutional BERT, we conducted an element-wise addition between the positional encoding and the feature maps obtained from 3DCNN to obtain the new tensor F ∈ R N×H ′ ×W ′ ×D : F n = S n + P n , n ∈ {1,. , N}. (6) In this study, we only used information about the position of the image in a sequence. While the observation date could be used as an alternative to the position, this would require adjusting the positional encoding function. Our experiments on the SDSS dataset did not reveal any improvement in the model when using the observation date, as opposed to just using the position. This can be understood because we do the training and the test with the same observation sequence and the network can therefore learn this sequence.
Evaluation metrics Accuracy is the probability that an object will be correctly classified. It is defined as the sum of the true positives plus true negatives divided by the total number of individuals tested: Accuracy = TP + TN TP + TN + FP + FN , (10) where TP, TN, FP, and FN are, respectively, the true positive, true negative, false positive, and false negative. The F1 score is a classification accuracy metric that combines precision and recall. It is a suitable measure of models tested with imbalanced datasets: Precision = TP TP + FP , (11) Recall = TP TP + FN , (12) F1 = 2 × Precision × Recall Precision + Recall. ( 13 ).
Implementation details The supernovae in our data are not all spectroscopically confirmed, which means that the unconfirmed ones might contain some misclassified objects due to errors from the photometric typing. The model may not generalize due to this data bias. To ensure that our model performs a generalization only on spectroscopically confirmed data, we split up the training process into two steps. We divided the data into two datasets. The first one contains only the photometrically typed data and the second contains spectroscopically confirmed data. We trained the model at first with the photometrically typed data, then we used transfer learning to fine-tune the model on only spectroscopically confirmed data (Table  3  summarizes the partition of the data). The models are trained using cross-validation of five folds and three ensembles in each fold.
All the architectures presented in this paper follow this same process and are implemented using PyTorch  (Paszke et al. 2019). We performed an extensive hyperparameter tuning of over 20 models to specify the best hyperparameters for our architecture, which contains 1. 3 Million parameters. We conducted a hyperparameter optimization using only a non-confirmed dataset with different parameters, such as sequence length, M, learning rate, lr, 3DCCN sub-sequence length, K, classifier layers' size, number of ConvBERT layers, L, number of Multi-head ConvAttention, T , batch size, and dropout. We used an Adam optimizer  (Kingma & Ba 2014) , with a value of the learning rate of 10 -3 , and we trained the model with cross-entropy loss and a dropout of 0. 3. Hyperparameter tuning involves the number of images K that feed the 3DCNN and the maximum length of the sequence.
Results This section provides studies on SDSS comparing the accuracy and F1 score of our proposed solution with other works. Table  4  summarizes the result of different models from different deep learning areas to diversify our benchmark as it contains RNN architectures (SuperNNova, LSTM), CNN-based models such as SCONE, Hybrid models that have  CNN and RNN such as Carrasco-Davis et al. (2019)  and  Gómez et al. (2020) , and, finally, a transformer-based model. Also, we compared the result using two types of datasets: first, the image dataset and, second, the same dataset object but with the light 1 https://github. com/DaBihy/ConvEntion curves; the goal is to highlight the advantage of using images instead of light curves. Moreover, the different works mentioned in Table  4  were initially proposed for different datasets with different classes and training protocols.
Hence, the results do not reflect the quality of these works on other datasets. The goal of the comparison is to give visibility into the performance of our model from a deep learning standpoint and the importance of using image time series from an astronomy perspective. Overall, our model ConvEntion obtains the highest accuracy of 79. 83% and F1 score of 70. 62%, 13 points higher in accuracy than the best results on images by  Gómez et al. (2020)  and 12 points higher in accuracy than the best model using light curves. This confirms the advantage of using images over light curves. This advantage can be explained by the fact that the image contains more information than a single value of flux in a light curve. Hence, a model can learn robustly with the existence of more high-level feature maps.
Also, the integration of a spatiotemporal feature extraction helped in getting a better high-level representation of the sequence, in comparison to separating the spatial features from the temporal ones. The two types of features have correlations that may help the model to better discriminate between objects. We can also highlight the importance of separating the band to mitigate the impact of missing observations. Our model performed well, in comparison to that of  Gómez et al. (2020)  which uses multiple bands, which shows that separating the bands and adding band embedding works better than feeding the network with empty bands. In the study of  Carrasco-Davis et al. (2019) , the authors trained their model on a dataset that only has a "g" band and they noted that the model can be adapted to classify the image sequence combining information using multiple bands.
Additionally, with a small dataset like ours, it is normal to have such behavior because the model does not have enough samples to totally discriminate among objects. Meanwhile, variables were  the best-classified class in our dataset, with just a bit of confusion with the AGN; this misclassification between AGN and variable can be explained by the class imbalance in our dataset based on the knowledge that the number of variables is higher than in the other classes. Table  5  summarizes the results of different models trained only on three classes (AGN, SN, Variable), where classes SNIa A141, page 9 of 10 A&A 673, A141  (2023)  and SNOther are combined into a single class. The goal of this experiment is to see the behavior of our model in discriminating between transient and non-transient objects.
We got the best results with an accuracy of 83. 90% with an F1 Score of 75. 77%. The model was able to classify the SN accurately, with a score of 86% (as shown in Fig. 7 ). The model is able to effectively process a given survey without any loss in performance and without the requirement of providing it with the time information for each image. However, when there is a covariate shift, or a mismatch, between the training set and the test set as when using a different dataset with a different observation sequence), incorporating the time information can improve the results. This experimental finding will be further studied and reported in future work using other datasets.
Conclusion In this work, we present a method for efficient astronomical image time series classification that is entirely based on the combination of convolutional networks and transformers. Inspired by action recognition and satellite image time series classification, we propose a model ConvEntion that utilizes convolutions and transformers jointly to capture complex spatio-temporal dependencies between distinct steps, leading to accurate predictions based on different observations of an object. The accuracy of our model is better with a high margin of 13%, in comparison to state-of-the-art methods using image data -and even better compared to approaches using light curves. Our model achieves good results on the SDSS dataset, while also being faster thanks to using fewer parameters and parallel computational processes, making it a good candidate for latencysensitive applications such as the real-time thumbnail classifier of astronomical events.
Acknowledgements. This work has been carried out thanks to the support of the  DEEPDIP ANR  project. This work makes use of Sloan Digital Sky Survey (SDSS ) data. Funding for SDSS-III has been provided by the  Alfred P. Sloan Foundation , the Participating Institutions, the  National Science Foundation , and the  U. S. Department of Energy Office of Science. The SDSS-III website is http://www. sdss3. org/. SDSS-III is managed by the  Astrophysical Research Consortium  for the Participating Institutions of the  SDSS-III Collaboration  including the  University of Arizona , the  Brazilian Participation Group ,  Brookhaven National Laboratory, Carnegie Mellon University, University of Florida , the  French Participation Group , the  German Participation Group, Harvard University , the  Instituto de Astrofisica de Canarias , the  Michigan State/Notre Dame/JINA Participation Group ,  Johns Hopkins University, Lawrence Berkeley National Laboratory, Max Planck Institute for Astrophysics, Max Planck Institute for Extraterrestrial Physics, New Mexico State University, New York University, Ohio State University, Pennsylvania State University, University of Portsmouth, Princeton University, the Spanish Participation Group, University of Tokyo, University of Utah, Vanderbilt University, University of Virginia, University of Washington , and  Yale University.
In section 4 the method is extended to larger systems of spin-1 {2 particles and explicitly worked out in the case of bipartite and tripartite systems, demonstrating the applicability of the path-sum formalism to diverse Hamiltonian structures. Finally, in section 5 we present numerical results and assess the computational performances of Matlab codes implementing the path-sum formalism, using 6-parameter chirped pulses as exemplar waveforms, in the context of nuclear magnetic resonance (NMR) and the presence of scalar interactions.
Numerical computations A Matlab implementation of the path-sum approach, presented in this work, has been made freely available  [22]. Here we present an assessment of its computational performances compared with the standard piecewise-constant propagator approximation, based on the identity U ptq " lim ∆tÑ0 N"t{∆t ź k"0 eiH pk∆tq∆t , (28) and adaptive Runge-Kutta method, using ode45 in Matlab.
Performances Throughout this section we consider a βptq for a chirped pulse as described in Section 5. 1. All computational tests take place on a Dell Laptop running Ubuntu 18. 04 equipped with Intel Core i7-8665U CPU @ 1. 90GHz 8 running Matlab R2019a. In Tables  1 to 3  we show the time and number of points N needed for each method to reach a target relative error E M of 10 ´3, 10 ´6 and 10 ´8. In the cases of Path-Sum (PS) codes, we have always set N p " N and N I " 1 to facilitate comparisons. These tests show that for relative errors E M on the order of 10 ´3 to 10 ´7-largely sufficient to simulate actual experimentspath-sum codes outperform the piecewise-constant propagator approximation (see in particular Figure  8 ) and even ode45. The latter observation is surprising given that: i) hear we are dealing with small systems for which ode45 is highly optimized; ii) ode45 is allowed to choose the number and position of its computation points while path-sum is constrained by our implementation to work on equally spaced points-a known disadvantage; iii) ode45's computation time degrades rapidly when asked to produce the solution at a large number of evaluation points.
Acknowledgements PLG acknowledges funding from the  Agence Nationale de la Recherche , grants  ANR-19-CE40-0006  and  ANR-20-CE29-0007. MF is grateful to the  Royal Society  for a  University Research Fellowship  and a  University Research Fellow Enhancement Award  (grant numbers  URF\R1\180233  and  RGF\EA\181018 ). We thank  E. Baligacs  for an improvement to the Matlab codes and for suggesting the Frobenius scalar product to evaluate accuracy.
The considered system is seen in nowadays applications. In  [2]  for example, it is stated that the considered structure (i. e. , the MAMRN) is the main topology structure for UAV cooperative surveillance networks. However, it is seen in  [10]  that the capacity region of the general MAMRN is still unknown. In MAMRN, the multiple access can be either orthogonal (as considered in this paper and in other works:  [16] ) or non-orthogonal (as in  [15] ), where orthogonality may be achieved through time, frequency, or code division multiplexing. Throughout the different prior arts that targeted the MAMRN, two major (recent) works are the ones done in the theses  [14]  and  [15]. In  [15] , the outage analysis of different examples of MAMRN is presented (check  [17] [18] [19] ). The analysis covers different coding and decoding schemes as well as different transmission scenarios (orthogonal and non-orthogonal).
Nevertheless, there is no decisive conclusion in this research area, and rigorous fair comparisons still have to be done for the slow fading half-duplex MAMRN. We summarize the mentioned relaying protocols in Table  2. Additionally, we assume that joint encoding (network coding) is performed on successfully decoded messages at the relaying nodes. Introduced in  [41]  for graphical networks, that concept implies that the relaying nodes not only can route or replicate the information received on input links to output links, but they can also encode that information before performing the transmission. The application of network coding to the physical layer of wireless networks is studied in various contributions, with particular attention to the combination of network and channel coding. Example applications in Table  2  Relaying protocols (RP) summary.
In Fig. 3a , we consider a (3, 2, 1)-MAMRN. At the considered time slot (any time slot in the retransmission phase), the decoding sets of all the nodes are presented. We see that the destination decoded the message of source s 1 , the sources did not decode any source message (but their own message), and the relays r 1 and r 2 decoded respectively the set of sources {s 1 } and {s 1 , s 2 , s 3 }. In  [98] , the candidate relaying nodes to be selected are the nodes that can help at least one source which is not decoded by the destination. As the destination decoded source s 1 , the candidate nodes (in this example) are the relaying nodes that can help either source s 2 or s 3 or both. Following the toy example, the candidate relaying nodes are s 2 , s 3 , and r 2 (highlighted in Fig. 3b ). After fixing the set of candidate relaying nodes, the destination chooses the node with the highest mutual information.
In this paper we present the first formally verified implementations of Kyber and, to the best of our knowledge, the first such implementations of any post-quantum cryptosystem. We give a (readable) formal specification of Kyber in the EasyCrypt proof assistant, which is syntactically very close to the pseudocode description of the scheme as given in the most recent version of the NIST submission. We present high-assurance open-source implementations of Kyber written in the Jasmin language, along with machine-checked proofs that they are functionally correct with respect to the EasyCrypt specification. We describe a number of improvements to the EasyCrypt and Jasmin frameworks that were needed for this implementation and verification effort, and we present detailed benchmarks of our implementations, showing that our code achieves performance close to existing hand-optimized implementations in C and assembly.