Reviewerly/exHarmony huggingface dataset:
Laypa: A Novel Framework for Applying Segmentation Networks to Historical Documents

We present novel software to process scans of historical documents to extract their layout information. We do this using a ResNet backbone with a feature pyramid head. We extract region information directly into PageXML. For baseline extraction, we use a two stage processing approach. The software has been applied successfully to several projects. The results show the feasibility to automatically label text lines and regions in historical documents.
Drawing the Line: A Dual Evaluation Approach for Shaping Ground Truth in Image Retrieval Using Rich Visual Embeddings of Historical Images

Images contain rich visual information that can be interpreted in multiple ways, each of which may be correct. However, current retrieval systems in computer vision predominantly focus on content-based and instance-based image retrieval, while other facets relevant to the querying person, such as temporal aspects or image syntax, are often neglected. This study addresses this issue by examining a retrieval system in a domain-specific document processing pipeline. A retrieval evaluation dataset, which focuses on the aforementioned tasks, is utilized to compare different promising approaches. Subsequently, a qualitative study is conducted to compare the usability of the retrieval results with their corresponding metrics. This comparison reveals a discrepancy between the best-performing model by performance metrics and the model that provides better results for answering potential research questions. While current models such as DINO and CLIP demonstrate their ability to retrieve images based on their semantics and contents with high reliability, they exhibit limited capabilities in retrieving other facets.
Two-step sequence transformer based method for Cham to Latin script transliteration

Fusion information between visual and textual information is an interesting way to better represent the features. In this work, we propose a method for the text line transliteration of Cham manuscripts by combining visual and textual modality. Instead of using a standard approach that directly recognizes the words in the image, we split the problem into two steps. Firstly, we propose a scenario for recognition where similar characters are considered as unique characters, then we use the transformer model which considers both visual and context information to adjust the prediction when it concerns similar characters to be able to distinguish them. Based on this two-step strategy, the proposed method consists of a sequence to sequence model and a multi-modal transformer. Thus, we can take advantage of both the sequence-to-sequence model and the transformer model. Extensive experiments prove that the proposed method outperforms the approaches of the literature on our Cham manuscripts dataset.
A hybrid CNN-Transformer model for Historical Document Image Binarization

Document image binarization is one of the main preprocessing steps in document image analysis for text recognition. Noise, faint characters, bad scanning conditions, uneven lighting or paper aging can cause artifacts that negatively impact text recognition algorithms. The task of binarization is to segment the foreground (text) from these degradations in order to improve optical character recognition (OCR) results. Convolutional Neural Networks (CNNs) are one popular method for binarization. But they suffer from focusing on the local context in a document image. We have applied a hybrid CNN-Transformer model to convert a document image into a binary output. For the model training, we used datasets from the Document Image Binarization Contests (DIBCO). For the datasets DIBCO-2012, DIBCO-2017 and DIBCO-2018, our model outperforms the state-of-the-art algorithms.
Document Layout Analysis with Deep Learning and Heuristics

The automated yet highly accurate layout analysis (segmentation) of historical document images remains a key challenge for the improvement of Optical Character Recognition (OCR) results. But historical documents exhibit a wide array of features that disturb layout analysis, such as multiple columns, drop capitals and illustrations, skewed or curved text lines, noise, annotations, etc. We present a document layout analysis (DLA) system for historical documents implemented by pixel-wise segmentation using convolutional neural networks. In addition, heuristic methods are applied to detect marginals and to determine the reading order of text regions. Our system can detect more layout classes (e.g. initials, marginals) and achieves higher accuracy than competitive approaches. We describe the algorithm, the different models and how they were trained and discuss our results in comparison to the state-of-the-art on the basis of three historical document datasets.
Gauging the Limitations of Natural Language Supervised Text-Image Metrics Learning by Iconclass Visual Concepts

Identification of images that are close to each other in terms of their iconographical meaning requires an applicable distance measure for text-image or image-image pairs. To obtain such a measure of distance, we finetune a group of contrastive loss based text-to-image similarity models (MS-CLIP) with respect to a large number of Iconclass visual concepts by means of natural language supervised learning. We show that there are certain Iconclass concepts that actually can be learned by the models whereas other visual concepts cannot be learned. We hypothesize that the visual concepts that can be learned more easily are intrinsically different from those that are more difficult to learn and that these qualitative differences can provide a valuable orientation for future research directions in text-to-image similarity learning.
Investigations on Self-supervised Learning for Script-, Font-type, and Location Classification on Historical Documents

In the context of automated classification of historical documents, we investigate three contemporary self-supervised learning (SSL) techniques (SimSiam, Dino, and VICReg) for the pre-training of three different document analysis tasks, namely script-type, font-type, and location classification. Our study draws samples from multiple datasets that contain images of manuscripts, prints, charters, and letters. The representations derived via pre-text training are taken as inputs for k-NN classification and a parametric linear classifier. The latter is placed atop the pre-trained backbones to enable fine-tuning of the entire network to further improve the classification by exploiting task-specific label data. The network’s final performance is assessed via independent test sets obtained from the ICDAR2021 Competition on Historical Document Classification. We empirically show that representations learned with SSL are significantly better suited for subsequent document classification than features generated by commonly used transfer learning on ImageNet.
An Evaluation of Handwritten Text Recognition Methods for Historical Ciphered Manuscripts

This paper investigates the effectiveness of different deep learning HTR families, including LSTM, Seq2Seq, and transformer-based approaches with self-supervised pretraining, in recognizing ciphered manuscripts from different historical periods and cultures. The goal is to identify the most suitable method or training techniques for recognizing ciphered manuscripts and to provide insights into the challenges and opportunities in this field of research. We evaluate the performance of these models on several datasets of ciphered manuscripts and discuss their results. This study contributes to the development of more accurate and efficient methods for recognizing historical manuscripts for the preservation and dissemination of our cultural heritage.
Nighttime Rainy Season Traffic Analysis: Vehicle Detection, Tracking, and Counting with YOLOv8 and DeepSORT

This research focuses on developing a reliable computer vision system for accurately tracking traffic density in India during the rainy season. The system uses deep learning-based techniques to handle the difficulties associated with vehicle detection and tracking. The three modules are vehicle detection, tracking, and vehicle counting. Vehicles are initially identified using the YOLOv8 algorithm, a state-of-the-art deep learning detector. Subsequently, the DeepSORT algorithm is utilized for multi-object tracking to ensure accurate and robust tracking of various objects, including cars, buses, trucks, bikes, and pedestrians. The importance of accurate vehicle counting and speed measurement is emphasized, especially during bad weather. An independently compiled dataset of Indian rainy conditions is used to assess the proposed computer vision system. The outcomes demonstrate the system's capability to accurately identify, track, count, and estimate the speeds of vehicles. These features offer insightful information for traffic analysis, including flow monitoring, congestion detection, and other associated traffic challenges. This study makes a contribution to the field of computer vision-based traffic monitoring and offers potential applications in transportation management systems under challenging weather conditions.
Model of trusted interaction of agents in decentralized cyber-physical environmentv

Objective . The purpose of the work is to increase the efficiency of the functioning of agents of a cyber-physical system in the presence of agents with incorrect or malicious behavior. The goal is achieved by establishing a trusted interaction between agents and quick detection of malicious impact. Method . Trusted interaction is carried out using distributed ledger technology and agent confidence indicators. The novelty of the proposed solution lies in the fact that information about the actions of agents is stored and aggregated using smart contracts at specified time intervals. Each agent keeps a local copy of the agent interaction chain. If several agents interact with each other, then they exchange information stored in their copies of the ledger. Result . To test the proposed method, we implemented it in C++. For the experiments, the scenario of collective perception by agents of a decentralized cyber-physical environment in a specialized simulation program Contiki Cooja was used. Conclusion . The method implemented using the solutions proposed in this work showed higher efficiency than the method based on the dynamic calculation of the confidence index. The proposed solutions can be applied not only in cyber-physical systems, but also in any other systems with decentralized control.
Volume‐based subset selection

Abstract This paper provides a fast algorithm for the search of a dominant (locally maximum volume) submatrix, generalizing the existing algorithms from to submatrix columns, where is the number of searched rows. We prove the bound on the number of steps of the algorithm, which allows it to outperform the existing subset selection algorithms in either the bounds on the norm of the pseudoinverse of the found submatrix, or the bounds on the complexity, or both.
Editorial. Using an AI interface to enhance reader experience: a test case

A s artificial intelligence (AI) systems have become more widely available, scientific publishers are expressing concern with their use in both manuscript development and peer review. [1]2][3] As publishers begin to establish guidelines for producing accurate and trustworthy manuscripts in the age of AI, 4 issues of copyright and intellectual property must be considered: any data uploaded into a public AI are ingested and "remembered" for others to potentially use as their own.AI "hallucinations" FIG. 1.
Role of Audio In Video Summarization

Video summarization attracts attention for efficient video representation, retrieval, and browsing to ease volume and traffic surge problems. Although video summarization mostly uses the visual channel for compaction, the benefits of audio-visual modeling appeared in recent literature. The information coming from the audio channel can be a result of audio-visual correlation in the video content. In this study, we propose a new audio-visual video summarization framework integrating four ways of audio-visual information fusion with GRU-based and attention-based networks. Furthermore, we investigate a new explainability methodology using audio-visual canonical correlation analysis (CCA) to better understand and explain the role of audio in the video summarization task. Experimental evaluations on the TVSum dataset attain F1 score and Kendall-tau score improvements for the audio-visual video summarization. Furthermore, splitting video content on TVSum and COGNIMUSE datasets based on audio-visual CCA as positively and negatively correlated videos yields a strong performance improvement over the positively correlated videos for audio-only and audio-visual video summarization.
Research on Sketching Face Headshot Generation based on Improved CycleGAN

At present, sketch heads generated from realistic heads still has problems such as blurred contours and missing textures. For this reason, this work proposes a sketch head generation method based on CycleGAN. Firstly, the Self-Attention Mechanism (Squeeze-and-Excitation Networks (SENet) module is added to the UNet self-encoder; secondly, the base model is transformed into a supervised learning model so as to add constraints on the generated avatars and the real avatars. The experimental results show that the sketched avatar generated by the method in this paper has a better visual effect on the CUHK student test set with a 0.0274 improvement in SSIM value than the sketched avatar generated by the base model.
Cross-Domain Similarity in Domain Adaptation for Human Activity Recognition

Human Activity Recognition (HAR) is a difficult machine learning problem, even for state-of-the-art deep learning models, due to HAR data's within-domain and cross-domain heterogeneity. Our research addresses the challenge of closed-set domain adaptation in heterogeneous, parameter-based, and transductive transfer learning on HAR datasets. We use a Bidirectional Long Short Term Memory (BLSTM)-based model that, in addition to training for classification accuracy using only labeled data from the source domain, also jointly trains on source and unlabeled target datasets to reduce the discrepancy between source and target domains using cross-domain similarity as an additional loss function. Our work contributes to existing research in the area of domain adaptation for HAR by evaluating the performance of the following cross-domain similarity metrics as loss functions in improving model classification accuracy: 1) Maximum Mean Discrepancy (MMD), which uses feature means to measure similarity between two domains; 2) Kernel Canonical Correlation Analysis (KCCA), which utilizes canonical correlations for similarity determination; and 3) Cosine Similarity, a metric that uses the cosine of the angle between two vectors as similarity measure. Our results demonstrate that MMD as a cross-domain similarity metric not only outperforms KCCA and Cosine Similarity in domain adaption, but also results in mean F1 score improvement of 45% over results where a model is trained solely on the target dataset.
A Question Answering System for Situation Puzzle with SPQA

There are many questions answering (QA) system built for solving QA tasks. In 2020 and 2022, Allen Institute and the University of Washington proposed UnifiedQA and UnifiedQA-v2. Their core concept is that the semantic understanding and reasoning capabilities required by models are common, and may not require format specific models although the QA task forms are different. Behind this concept, I build a new QA model named SPQA, aiming to answer the situation puzzle questions by adding new situation-puzzle related dataset (SpQ). In addition, I evaluate the performance of SPQA and UnifiedQA-v2 for fine-tuning and prompt-tuning. The results of fine-tuning indicate that SpQ dataset is important for fine-tuning and prompt-tuning to answer situation puzzle questions well, but also make the answering ability of normal yes/no questions worse. Eventually, the results of prompt-tuning indicate that the effects of SpQ is larger and more significant on situation puzzle questions and normal yes/no questions under the same data scale. In the future work, the further research like building larger SpQ dataset should be considered.
Graph Pruning and Representation Learning for Stance Detection

Stance detection has attracted wide attention, especially on social media platforms, which is particularly challenging. It can be regarded as a short text classification task, which aims to identify the author's stance (Favor, Against, or None) expressed within the text towards a specific target. Most of the existing research based on the pre-training language model ignores global word co-occurrence with discontinuous and long-distance semantics in the corpus itself. To overcome such drawbacks, we propose a graph neural network framework that integrates semantic features and structural features to improve the performance of stance detection. Through the fine-tuning of the pre-training model on the task of stance detection, and graph pruning and sampling in graph pre-training model, the representation of text nodes takes into account both global structure and semantic information. Experimental results on the public dataset SemEval-2016 show that our model outperforms other baseline methods. In addition, the ablation experiments demonstrates the effectiveness of each part of our proposed method.
A Class-Added Continual Learning Method for Motor Fault Diagnosis Based on Knowledge Distillation of Representation Proximity Behavior

Continual learning is promising in intelligent motor fault diagnosis because it enables networks to increase diagnosable fault classes without time-consuming retraining during new fault happening. However, the traditional continual learning based on knowledge distillation keeps the absolute positions of samples in representation spaces to prevent catastrophic forgetting, which limits new fault samples to embedding into representation spaces flexibly. To address this issue, a continual learning method based on a novel knowledge distillation strategy is proposed for motor fault diagnosis. At incremental stages of continual learning, new and old diagnosis networks are first regarded as the teacher and student networks. Then, the improved distillation strategy is designed to guide knowledge transfer from teacher networks to student networks, meanwhile, student networks learn from the new fault samples. Finally, new diagnosis networks are obtained which can diagnose incremental fault classes. For the improved knowledge distillation strategy, knowledge is inherited by maintaining the proximity behavior of samples in the representation spaces, thereby networks can learn to map samples into representation spaces more flexibly. Through a study case of class-added fault diagnosis of motors, it is proved that the proposed method can improve diagnostic accuracy during continual learning.
TEMI-MOT: Towards Efficient Multi-Modality Instance-Aware Feature Learning for 3D Multi-Object Tracking

3D multi-object tracking is one of the key technologies of autonomous driving, which aims to ensure that autonomous driving vehicles accurately perceive the movements and intentions of surrounding traffic participants. In recent years, some 3D multi-object tracking methods based on multimodality have been proposed. Although these methods improve the accuracy of object association in the tracking process, these methods are still difficult to effectively deal with the problems of feature ambiguity due to occlusion, incorrect feature alignment between different modalities, and confusion of adjacent target features caused by coarse-grained feature maps. To address these problems, we propose a new multi-modality feature learning method for 3D multi-object tracking, named TEMI-MOT, which is composed of three modules in series: the point-guided image feature sampler, the instance-aware feature encoder, and the tracking pipeline. The point-guided feature sampler realizes the alignment between the point cloud and image features, the instance-aware feature encoder fuses the aligned image features with each object's points to generate the discriminative instance-aware features, and the tracking pipeline finally outputs the results based on instance-aware features and G-IoU geometric similarities. Our approach achieves state-of-the-art results on the nuScenes dataset among the methods using CenterPoint detections. The experimental results show that the proposed method has better robustness and effectiveness for 3D multi-object tracking.
Deep Learning-Based Modulation Classification for OFDM Systems Without Symbol-Level Synchronization

Deep learning (DL)-based modulation classification of incoherently received orthogonal frequency division multiplexing (OFDM) signals is studied. We propose a novel preprocessing algorithm to build features characterizing the modulation of OFDM signals, which are insensitive to synchronization error. With obtained features, pilot subcarrier indices used for CFO correction may also be estimated. The features obtained with the proposed algorithm are classified with a convolutional neural network (CNN)-based classifier. We have evaluated classification performance with simulated and hardware-generated data. Using these features, the modulation classifier outperforms existing DL-based classifiers which assume symbol-level synchronization with up to 25% classification accuracy performance gain.
A Novel Sequential Recommendation Model Based on the Filter and Model Augmentation

The sequential recommendation aims to capture the user's dynamic interest characteristics according to the historical interaction sequences of users, to provide users with much more personalized recommendations. Although the popular transformer-based sequential recommendation models can effectively capture items dependencies between sequences, they also have some problems such as the noisy information and data sparseness in reality. Therefore, this paper proposes a novel sequential recommendation model based on Filter and Model Augmentation Strategies, named FMAS in short. In particular, during the embedding process, the proposed FMAS utilizes the Fast Fourier Transform (FFT) to convert the original user sequence embedding into the style of the frequency domain, which can be filtered the noise information via a learnable filter without destroying the sequence correlations; Moreover, to alleviate the problem of data sparsity, the FMAS proposes two model augmentation strategies, named Layer Drop and neural masking, to construct contrastive learning views. Experimental results demonstrate that the FMAS model outperforms many sequential recommendation approaches.
Fuzzy Networks for Explainable Artificial Intelligence

Advanced machine learning techniques are very powerful in predictive tasks. However, they are mostly weak in explaining the inference process and they are mostly treated as black-box models. Fuzzy Network (FN) is powerful white-box technique which is capable of dealing with complexity and linguistic uncertainty. In this paper, a method is introduced to optimise Rule Based Networks using Fuzzy C-Means (FCM) for rule reduction, Genetic Algorithms to tune the membership functions and Backward Selection to reduce the inputs and network branches. A case study in transport and telecommuting is used to illustrate the performance of the proposed method. The results show the FN ability to explain the internal process of decision making and its capabilities in transparency and interpretability as an Explainable AI method.