dataesr/scientific-paragraphs-categorization huggingface dataset:
Acknowledgements The authors acknowledge support from the  HPC  computing centres and resources hpc-qcd (CERN),  HPE Apollo Hawk (HLRS)  under the grant number  stabwf/44185 ,  Cori (NERSC) ,  Frontera (TACC) ,  Piz Daint  ( CSCS ) as well as Occigen (CINES),  Jean-Zay  (IDRIS) and  Irène-Joliot-Curie (TGCC)  under projects ( 2020 , 2021,2022)-A0080511504  and (2020,2021,2022)-A0080502271 by  GENCI  as well as project 2021250098 by  PRACE. AS  acknowledges funding support under the  National Science Foundation  grant  PHY-2209185. AF acknowledges support by the  Ministry of Science and Technology Taiwan (MOST)  under grant  111-2112-M-A49-018-MY2. We greatly acknowledge the leading role of  Martin Lüscher  during the development and implementation of the SWF framework.
1. OMCBIR builds a lightweight convolutional neural network as a feature extractor to obtain image features, which is the core part of the framework. With the help of OMCBIR, we can implement offline image retrieval on terminal devices. However, the most important factor affecting retrieval performance is feature extraction, so we focus on improving this module. Although some lightweight CNNs have been proposed, they still suffer from a large amount of expensive computing power and memory usage. To further reduce the model parameters and computational complexity, we propose an Attention-based Lightweight Block (ALBlock) unit. ALBlock significantly reduces the amount of computation and parameters within the block by grouping a large number of 1 × 1 convolutions in the bottleneck block  [3].
(2) As the core of OMCBIR, we propose an extremely lightweight network architecture ALNet, whose role is to extract features from images. Besides, we also propose an ALBlock unit, which is the basic building block of ALNet. By introducing pointwise group convolution, channel shuffle and convolution attention module in bottleneck block, we not only improve retrieval accuracy, but also greatly reduce the computational cost and memory usage of the model. (3) The experimental results on five public datasets demonstrate the effectiveness of our model. Compared to MobileNetV2, ALNet obtains a higher mAP on each dataset in OMCBIR with over 62% reduction in model parameters and over 63% reduction in model size.
Deep learning based online image retrieval In the last decade, following the emergence of deep learning, feature representation begin to shift from hand-crafted to learning-based  [4] , in which feature learning based on CNNs replaces traditional hand-crafted feature representation as the most advanced pipeline. Deep learning is a hierarchical feature representation technology used to learn abstract features that are important to datasets and applications from data. At present, CNN features have been used as a general expression for content-based image retrieval, and a lot of researches are devoted to obtain image features that are conducive to retrieval for online image retrieval. For example, previous work  [5]  optimizes AlexNet from the pooling layer, fully connected layer, and hidden layer, and obtains state of the art results on large-scale image retrieval tasks.
Kalantis et al. [7]  mainly propose a direct and effective image representation method based on the cross-dimensional weighting and aggregation of the output of the deep convolutional neural network layer. Tolias et al. [8]  construct a compact feature vector to encode multiple image regions without providing multiple inputs to the neural network, and proposes an R-MAC pool method for image retrieval. With the popularity of Generative Adversarial Networks  [11, 12]  (GANs), many researchers have explored the possibility of using GAN in image-related applications because it shows great potential. Some of the existing research methods tend to use GAN to obtain more training data. By using GAN to generate synthetic images that are visually similar to the input image, this is conducive to image retrieval tasks, because the ultimate goal is to retrieve similar images from a given dataset.
The basic idea of Xception is to design a channel-separated convolution. Because there are a large number of dense 1 × 1 convolutions in Xception, the network is very inefficient. Therefore, ShuffleNet network that combines pointby-point group convolution and channel shuffle is proposed. Based on neural structure search technology, EfficientNet finds a simple and efficient composite coefficient to enlarge the network from three dimensions of depth, width and resolution to improve the accuracy of the model, which requires a lot of GPUs. MobileNet uses depth-wise separable convolution to build a lightweight deep neural network. On the basis of depth-wise separable convolution, MobileNetV2 introduces two modules: inverted residual and linear bottleneck, which makes the model smaller and faster when the network is deeper, but it still has a large amount of expensive point-by-point convolutions.
Methodology We propose an offline mobile content-based image retrieval (OM-CBIR) framework based on a lightweight neural network. In this framework, the most important factor affecting deployment and retrieval is the feature extraction module. In order to further compress the model and to obtain image features that are conducive to retrieval, we focus on the improvement of feature extraction module, and propose ALBlock unit and ALNet lightweight neural network to further reduce the computation and parameters of the model. The core details of this framework will be elaborated in the following sections.
We add another CBAM module after the 1 × 1 convolutional layer, and finally, replace the average pooling layer with a binary adaptive average pooling layer (AdaptiveAvgPool2d) to obtain image features of the same size for image retrieval. When the classification training is over, the desired model features is extracted by removing the classifier of the pre-trained model instances. The network structure of ALNet adopts a single stack mode, and the basic building block is ALBlock, so as to achieve the effect of reducing the amount of model calculation without losing the retrieval accuracy. X. Zhang et al.
Implementation details In the proposed OMCBIR, the model parameters of ALNet will be saved with the highest classification accuracy during the entire training process. When the training epoch reaches the preset value, we terminate the training. Before the model conversion, the classifier of the model is removed with the aim of obtaining the output features by forward inference of the model. After that, the model conversion provided by the Pytorch Mobile framework is used to complete the conversion of the PyTorch model to the TorchScript model, which supports the efficient operation of machine learning models on edge devices. Finally, we deploy the well trained model to OMCBIR, by using PyTorch Mobile's java interface to call the model in the Android project. In OMCBIR, we extract the image features of the image database in batches and store them in the local database.
Experiments We conduct experiments to evaluate the performance of OMCBIR framework. These experiments are tested on five widely used datasets. Classification accuracy is an important indicator for evaluating the network structure. Therefore, we firstly evaluate the classification accuracy, computation, parameter amount and model size of ALNet and other classical lightweight neural networks. In addition, we compare mAP of ALNet with other state-of-the-art algorithms in OMCBIR. Finally, the ablation experiments of different components are performed, using MobileNetV2 as a baseline.
Datasets MNIST  [35]  dataset is a very classic dataset in the field of image classification, consisting of 60,000 training samples and 10,000 test samples, each of which is a 28 × 28 pixel gray scale handwritten digital image. Since Pytorch Mobile does not support reading gray scale images, we use the method of overlaying channels for training. SVHN  [36]  (Street View House Number) dataset comes from the Google Street View house number and contains more than 600,000 digit images. The native dataset is a number of unprocessed color images with multiple digits on each image. This dataset contains PNG images and digitStruct. mat files. The. mat files are used in the experiments, and each image is 32 × 32 in size. When performing image retrieval,. mat file is needed to convert into an RGB image by using the image's filename as GroundTruth.
SUN397  [37]  dataset contains 108,753 images of 397 categories, used in the Scene UNderstanding (SUN) benchmark. The number of images varies across category, but there are at least 100 images per category. We process the image categories into labels, then training images and test images are combined, de-duplicated and sorted. Finally, the training images and test images are divided into 8:2 according to random grouping. After our processing, there are 76,872 training images and 19,218 test images, and the total dataset contains 96,090 images of 397 categories. Oxbuild5K  [38]  dataset contains 5063 images of 17 Oxford landmarks obtained from Flickr. We process the categories into labels, corresponding to 1-17 numbers respectively. We have preprocessed the image so that the image size for input model training and image retrieval is 160 × 160.
mAP represents the average percentage of similar images in all retrieved images after evaluating all queries. The formula for the average accuracy of  searches is calculated as follows: 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛@𝑘 = ∑ 𝑘 𝑖=1 𝑅 𝑒 (𝑖) 𝑘 ( 7 ) where   () ∈ [0, 1] indicates whether the current th image is an image with the same label and determines the correlation between the query image and the th image. 0 means that this image does not have the same label as the query image, and 1 means that this image has the same label. The average precision (mAP) formula is defined as follows: 𝑚𝐴𝑃 (𝑄) = 1 𝑄 𝑄 ∑ 𝑖=0 1 𝑚 𝑚 ∑ 𝑘=1 𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛@𝑘 ( 8 ) where  is the number of the set of query images, and  is the number of similar images in the dataset.
Experimental setting For the experiments, Pytorch is used to train the classification model and Pytorch Mobile framework is used to complete the deployment of the deep learning model on the mobile terminal. The experiments use the Stochastic Gradient Descent (SGD) optimization algorithm, and the weight decay is 1 × 10 -4. For datasets with smaller images such as MIN-IST and SVHN, batch size during training is 128, and 64 during testing. For other datasets with larger size of images, batch size during training is 32, and 16 during testing. The initial learning rate is 0. 1. According to the total number of training times, different epoch numbers are set to reduce the learning rate. In order to prove the effectiveness of ALNet, we have conducted comprehensive experiments on ALNet, ShuffleNet, ShuffleNetV2, MobileNets, MobileNetV2 and EfficientNet on the five public datasets.
Evaluation on model metrics In this subsection, we compare ALNet with classic lightweight neural network models (including ShuffleNet, MobileNets, ShuffleNetV2, MobileNetV2 and EfficientNet) in recent years. We conduct feature extraction training for the model oriented to classification tasks, so in this section we show the comparison of the Acc, FLOPs, Params, and Model Size of different algorithms on five datasets. As shown in Fig. 3 (a), (c ) and (d), the experimental comparison shows that the proposed ALNet is much smaller than other lightweight models in terms of Params and model size when there is little difference in classification accuracy, which can save a lot of memory footprint. It can be seen from Fig. 3(b ) that on all datasets, FLOPs of ALNet are much smaller than ShuffleNet, ShufflenetV2 and MobileNets.
Evaluation on retrieval task We use ALNet without the last layer of classifier as the feature extractor on the mobile terminal to test the retrieval performance of OMCBIR. For other popular lightweight CNN networks (including Shuf-fleNet, MobileNets, ShuffleNetV2, MobileNetV2 and EfficientNet), we use the same method to test retrieval performance on the mobile side. Moreover, we analyze the performance on each dataset and compared with other state-of-the-art methods. Finally, we conduct ablation experiments on the proposed ALNet in OMCBIR. The details of experiments are shown in the following.
Results on SVHN Table  3  summarizes the comparison results of each method on SVHN dataset. The proposed method obtains a mAP@1 of 82. 66%, second after ShuffleNetV2, but FLOPs, Params and Model Size are smaller than ShuffleNetV2, it is worth mentioning that the amount of calculation is only 14% of ShuffleNetV2. At the same time, compared with MobileNetV2, our method reduces FLOPs by 0. 7%, Params by 61. 6% and Model Size by 63. 4% when mAP@1 is higher than the baseline. The comprehensive advantages are more prominent.
Results on SUN397 Experiments on the large-scale scene understanding dataset SUN397 are conducted and the experimental results are shown in Table  4. Our proposed method obtains the highest mAP@5 of 95. 76%. Although slightly unsatisfactory on mAP@1, it is still higher than the baseline model. Compared with the first three methods, ALNet produces a better compression and acceleration rate than the ShuffleNet series and MobileNets, as shown in FLOPS, Param, and Model Size. Compared with MobileNetV2, although it is slightly higher in FLOPs, ALNet has advantages in other aspects. (In terms of mAP@1, they are 33. 67% and 33. 60%, in terms of parameters, they are 0. 77M and 2. 74M, and in terms of Model Size, they are 3. 4M and 9. 4M). Therefore, the proposed method demonstrates its high-performance retrieval capabilities and the ability to compress lightweight neural networks.
Results on Oxbuild5k and Paris6k Tables  5  and 6  summarize the comparison results on the Oxbuild5k and Paris6k datasets. The proposed method obtains 19. 91% mAP@1 and 93. 58% mAP@5 on Oxbuild5k, which is higher than other methods. In terms of Param and Model Size, the best results are also obtained, which are 0. 77M and 3. 4M respectively, which is convenient for deployment in mobile devices. We observe that ALNet always has a slightly higher calculation amount than MobileNetV2 on datasets with larger size of images (SUN397, Oxbuild5k, Paris6k), which is the weakness of our method, but it is better than MobileNetV2 in the other three indicators. It can be seen from Table  6  that the proposed method shows more advantages in compression rate and retrieval. Similar results are observed on Paris6K, the highest mAP@1 of 24.
Ablation study In order to evaluate the effectiveness of ALNet, we conduct detailed ablation studies on the five datasets. As shown in Table  7 , we use MobileNetV2 as the baseline and denote our model that uses grouped convolution and the new network structure as Lite-Network (LNet). On the basis of LNet, a lightweight network with channel shuffle (CS) and CBAM is the proposed ALNet. Table  7  reports the results of various parts of the ablation experiment of ALNet. Similar observations can be found in other datasets. Channel Shuffle: From the comparison of Params and Model Size of MobileNetV2 and LNet, it can be seen that grouping convolution can effectively reduce the Params and Model Size of the model, and the minimum values are obtained on all datasets. However, mAP is almost lower than the baseline on each dataset, so the grouping operation weakens the performance of the model to a certain extent.
Channel shuffle is proposed to solve the problem of feature communication between different groups. It can be seen from the third row of Table  7  that adding channel shuffle does not increase the parameters and size of the model, but improves the model retrieval performance to varying degrees. CBAM: Channel Attention Module and Spatial Attention Module perform attention on the channel and space respectively. It not only saves parameters and computing power, but also ensures that it can be integrated into the existing network as a plug-and-play module in the architecture. As shown in Table  7 , after adding channel shuffle, although the retrieval performance on the MINIST, SUN397 and Oxbuild5k datasets is better than the baseline, it is still slightly insufficient on SVHN and Paris6k. In order to improve the feature expression ability of the model, we introduce an attention mechanism to let the network learn to pay attention to key information.
From the fourth row of Table  7 , it can be seen that mAP is almost improved after adding CBAM with only a small increase in parameters and model size. By adding channel shuffle after the 1 × 1 group convolution and adding a convolutional attention mechanism after the depth separable convolution, we obtain the best model. The placement order of the above modules is the best result obtained through a large number of experiments. Comparing the rows in Table  7 , we can see that both channel shuffle and convolutional attention mechanisms can basically enhance the retrieval performance of the model. It can be concluded from the last row of Table  7  that under the combined effect of the two, our model has obtained the highest mAP that exceeds other algorithms on each dataset.
Conclusions This paper proposes an offline mobile retrieval framework called OMCBIR. In this framework, we use lightweight CNN to accomplish feature extraction in order to deploy the deep learning model on mobile device. To further reduce model redundancy, we redesign the block structure. By reconstructing the model structure and introducing the convolutional attention module, an extremely lightweight small network architecture ALNet is designed, which greatly reduces the computational cost and memory consumption of the model while improving the retrieval accuracy. After that, the model is used as a feature extractor, which is converted and deployed to Android platform to achieve image retrieval. A comprehensive experiments are carried out in the experimental verification part. We evaluate retrieval performance and model compression indicators of ALNet and the latest SOTA methods on different datasets.
OMCBIR's workflow. Input: training data X and training epoch ℎ. Output: The compact model and its parameters  *. Initialize the weights of model; 2: for ℎ = 1 → ℎ  do Update the model parameter W based on X; Calculate the prediction results of the model; Save the model weights with the highest accuracy; 6: end for 7: if training is over then Delete the classification layer of the model; Perform model conversion and save the. pt model; 10: end if 11: Implementation: 12: Load the. pt model on the Android platform; 13: Extract the features of the datasets images, and store them in the database; 14: Extract the features of the query image; 15: Calculate the distance and get similar pictures. 16: return mAP.
Algorithm 1 1 : 3 : 4 : 5 : 8 : 9 : 1134589 OMCBIR's workflow. Input: training data X and training epoch ℎ. Output: The compact model and its parameters  *. Initialize the weights of model; 2: for ℎ = 1 → ℎ  do Update the model parameter W based on X; Calculate the prediction results of the model; Save the model weights with the highest accuracy; 6: end for 7: if training is over then Delete the classification layer of the model; Perform model conversion and save the. pt model; 10: end if 11: Implementation: 12: Load the. pt model on the Android platform; 13: Extract the features of the datasets images, and store them in the database; 14: Extract the features of the query image; 15: Calculate the distance and get similar pictures. 16: return mAP.
VeriDevOps offers a methodology and a set of integrated mechanisms that significantly improve automation in DevOps to protect systems at operations time and prevent security issues at development time by (1) specifying security requirements, (2) generating trace monitors, (3) locating root causes of vulnerabilities, and (4) identifying security flaws in code and designs. This paper presents a methodology that enhances productivity and enables the continuous integration/delivery of trustworthy systems. We outline the methodology, its application to relevant scenarios, and offer recommendations for engineers and managers adopting the VeriDe-vOps approach. Practitioners applying the VeriDevOps methodology should include security modeling in the DevOps process, integrate security verification throughout all stages, utilize automated test generation tools for security requirements, and implement a comprehensive security monitoring system, with regular review and update procedures to maintain relevance and effectiveness.
INTRODUCTION In today's fast-paced software development world, the need for security is more crucial than ever before. However, security is often seen as an afterthought, and it is only addressed after an application has been deployed, making it difficult and costly to fix any issues. This is where DevOps  [9]  comes into play since it helps to combine the speed and agility of software development with the reliability and security of operations. SecDevOps  [11]  is a software development methodology that merges DevOps practices with security principles, aiming to establish a seamless and cohesive approach to software development and security. The primary objective of SecDevOps is to incorporate security measures at every phase of the software development lifecycle (SDLC), encompassing planning, design, deployment, and maintenance stages.
Through the seamless integration of security throughout the SDLC, SecDevOps empowers organizations to deliver secure software in a faster and more efficient manner. To integrate security into the DevOps process effectively, it is recommended to employ security verification, automated security verification, and security monitoring techniques. Security verification  [3]  is a method of demonstrating that a system or application is secure through assurance and reasoning. Automated security testing  [5]  involves using software tools to automatically generate and execute tests in order to detect any security vulnerabilities in an application. Finally, security monitoring  [13]  involves continuously tracking an application's performance and behaviour to detect any attacks or potential security breaches.
Together, these methods make it possible for the VeriDevOps methodology to provide a comprehensive approach to security in DevOps, enabling teams to identify and address security issues throughout the development process rather than after deployment. In this methodology, security is integrated into every stage of the DevOps pipeline, from planning and development to testing, deployment and monitoring. The VeriDevOps methodology combines multiple techniques to embed security throughout the DevOps lifecycle. It begins by using Natural Language Processing (NLP) to improve security requirements, eliminating poorly written specifications and enhancing their understandability. Modern NLP methods are further leveraged to extract, classify, and semantically search related security recommendations from natural language texts.
A requirements-ascode paradigm is implemented to encapsulate requirements and their verification methods within object-oriented classes, enabling executable requirements, traceability, and reuse. To detect security incidents, monitoring tools analyze network traffic and system logs. Real-time detection tools enhance this process, inspecting network traffic at the packet level with pre-trained neural network models for the swift identification of threats. At the development level, metamorphic testing is employed when systems lack explicit test oracles, assessing the impact of input transformations on outputs. Lastly, specific methodologies are used to undergo validation and generate regression test cases. Collectively, these techniques create a comprehensive, secure, and efficient DevOps process.