Reviewerly/exHarmony huggingface dataset:
Combining OCR Models for Reading Early Modern Books

In this paper, we investigate the usage of fine-grained font recognition on OCR for books printed from the 15th to the 18th century. We used a newly created dataset for OCR of early printed books for which fonts are labeled with bounding boxes. We know not only the font group used for each character, but the locations of font changes as well. In books of this period, we frequently find font group changes mid-line or even mid-word that indicate changes in language. We consider 8 different font groups present in our corpus and investigate 13 different subsets: the whole dataset and text lines with a single font, multiple fonts, Roman fonts, Gothic fonts, and each of the considered fonts, respectively. We show that OCR performance is strongly impacted by font style and that selecting fine-tuned models with font group recognition has a very positive impact on the results. Moreover, we developed a system using local font group recognition in order to combine the output of multiple font recognition models, and show that while slower, this approach performs better not only on text lines composed of multiple fonts but on the ones containing a single font only as well.
SIMARA: A Database for Key-Value Information Extraction from Full-Page Handwritten Documents

We propose a new database for information extraction from historical handwritten documents. The corpus includes 5,393 finding aids from six different series, dating from the 18th-20th centuries. Finding aids are handwritten documents that contain metadata describing older archives. They are stored in the National Archives of France and are used by archivists to identify and find archival documents. Each document is annotated at page-level, and contains seven fields to retrieve. The localization of each field is not available in such a way that this dataset encourages research on segmentation-free systems for information extraction. We propose a model based on the Transformer architecture trained for end-to-end information extraction and provide three sets for training, validation and testing, to ensure fair comparison with future works. The database is freely accessible ( https://zenodo.org/record/7868059 ).
ICDAR 2023 Competition on Visual Question Answering on Business Document Images

This paper presents the competition report on Visual Question Answering (VQA) on Business Document Images (VQAonBD) held at the 17th International Conference on Document Analysis and Recognition (icdar 2023). Understanding business documents is a crucial step toward making an important financial decision. It remains a manual process in most industrial applications. Given the requirement for a large-scale solution to this problem, it has recently seen a surge in interest from the document image research community. Credit underwriters and business analysts often look for answers to a particular set of questions to reach a decisive conclusion. This competition is designed to encourage research in this broader area to find answers to questions with minimal human supervision. Some problem-specific challenges include an accurate understanding of the questions/queries, figuring out cross-document questions and answers, the automatic building of domain-specific ontology, accurate syntactic parsing, calculating aggregates for complex queries, and so on. Further, despite having the same accounting fundamentals, the terminologies and ontologies used across different organizations and geographic locations may vary significantly. This makes the problem of generic VQA on such documents only more challenging. Since this is the first iteration of the competition, it was restricted in terms of some of the challenges listed; however, the further iterations of this competition aim to include many additional sub-tasks with the larger vision of accurate semantic understanding of business documents as images. Eleven different teams around the world registered for this competition. Five teams out of those submitted methods spanning multiple approaches, among which Team Upstage KR won the competition with a weighted average score of 95.9%. The runner-up team, NII-TablQA obtained a weighted average score of 90.1%
Topic Shift Detection in Chinese Dialogues: Corpus and Benchmark

Dialogue topic shift detection is to detect whether an ongoing topic has shifted or should shift in a dialogue, which can be divided into two categories, i.e., response-known task and response-unknown task. Currently, only a few investigated the latter, because it is still a challenge to predict the topic shift without the response information. In this paper, we first annotate a Chinese Natural Topic Dialogue (CNTD) corpus consisting of 1308 dialogues to fill the gap in the Chinese natural conversation topic corpus. And then we focus on the response-unknown task and propose a teacher-student framework based on hierarchical contrastive learning to predict the topic shift without the response. Specifically, the response at high-level teacher-student is introduced to build the contrastive learning between the response and the context, while the label contrastive learning is constructed at low-level student. The experimental results on our Chinese CNTD and English TIAGE show the effectiveness of our proposed model.
LineFormer: Line Chart Data Extraction Using Instance Segmentation

Data extraction from line-chart images is an essential component of the automated document understanding process, as line charts are a ubiquitous data visualization format. However, the amount of visual and structural variations in multi-line graphs makes them particularly challenging for automated parsing. Existing works, however, are not robust to all these variations, either taking an all-chart unified approach or relying on auxiliary information such as legends for line data extraction. In this work, we propose LineFormer, a robust approach to line data extraction using instance segmentation. We achieve state-of-the-art performance on several benchmark synthetic and real chart datasets. Our implementation is available at https://github.com/TheJaeLal/LineFormer .
Robustness Evaluation of Transformer-Based Form Field Extractors via Form Attacks

We propose a novel framework to evaluate the robustness of transformer-based form field extraction methods via form attacks. We introduce 14 novel form transformations to evaluate the vulnerability of the state-of-the-art field extractors against form attacks from both OCR level and form level, including OCR location/order rearrangement, form background manipulation and form field-value augmentation. We conduct robustness evaluation using real invoices and receipts, and perform comprehensive research analysis. Experimental results suggest that the evaluated models are very susceptible to form perturbations such as the variation of field-values ( $$\sim $$ 15% drop in F1 score), the disarrangement of input text order( $$\sim $$ 15% drop in F1 score) and the disruption of the neighboring words of field-values( $$\sim $$ 10% drop in F1 score). Guided by the analysis, we make recommendations to improve the design of field extractors and the process of data collection. Code will be available at here .
On Web-based Visual Corpus Construction for Visual Document Understanding

In recent years, research on visual document understanding (VDU) has grown significantly, with a particular emphasis on the development of self-supervised learning methods. However, one of the significant challenges faced in this field is the limited availability of publicly accessible visual corpora or extensive collections of images with detailed text annotations, particularly for non-Latin or resource-scarce languages. To address this challenge, we propose Web-based Visual Corpus Builder (Webvicob), a dataset generator engine capable of constructing large-scale, multilingual visual corpora from raw Wikipedia HTML dumps. Our experiments demonstrate that the data generated by Webvicob can be used to train robust VDU models that perform well on various downstream tasks, such as DocVQA and post-OCR parsing. Furthermore, when using a dataset of 1 million images generated by Webvicob, we observed an improvement of over 13% on the DocVQA Task 3 compared to a dataset of 11 million images from the IIT-CDIP. The implementation of our engine is publicly available on https://github.com/clovaai/webvicob .
WordStylist: Styled Verbatim Handwritten Text Generation with Latent Diffusion Models

Text-to-Image synthesis is the task of generating an image according to a specific text description. Generative Adversarial Networks have been considered the standard method for image synthesis virtually since their introduction. Denoising Diffusion Probabilistic Models are recently setting a new baseline, with remarkable results in Text-to-Image synthesis, among other fields. Aside its usefulness per se, it can also be particularly relevant as a tool for data augmentation to aid training models for other document image processing tasks. In this work, we present a latent diffusion-based method for styled text-to-text-content-image generation on word-level. Our proposed method is able to generate realistic word image samples from different writer styles, by using class index styles and text content prompts without the need of adversarial training, writer recognition, or text recognition. We gauge system performance with the Fréchet Inception Distance, writer recognition accuracy, and writer retrieval. We show that the proposed model produces samples that are aesthetically pleasing, help boosting text recognition performance, and get similar writer retrieval score as real data. Code is available at: https://github.com/koninik/WordStylist .
AFFGANwriting: A Handwriting Image Generation Method Based on Multi-feature Fusion

Recently, reliable quality images can be generated due to the development of adversarial generative networks. Nevertheless, computer-generated images are still not comparable to humans in terms of handwriting image generation. In this paper, a novel method (i.e. AFFGANwriting) based on multi-scale features fusion has been proposed for handwriting image generation. In AFFGANwriting, a style encoder based on VGG19 has been designed to extract features in different scales. In this way, a variety of global features (e.g. stroke thickness, inclination and so on) and local features (e.g. continuous strokes, personalized writing and so forth) can be obtained. After that, the global features and the local features can be fused together to generate much more realistic handwriting images by multiple feature fusion modules of AFFGANwriting. Experimental results demonstrate that the proposed method can be competent for the task of handwriting images generation and outperforms various baseline and state-of-the-art methods. The code is available at: https://github.com/wh807088026/AFFGanWriting .
Improved Learning for Online Handwritten Chinese Text Recognition with Convolutional Prototype Network

Segmentation-based handwritten text recognition has the advantage of character interpretability but needs a character classifier with high classification accuracy and non-character rejection capability. The classifier can be trained on both character samples and string samples but real string samples are usually insufficient. In this paper, we proposed a learning method for segmentation-based online handwritten Chinese text recognition with a convolutional prototype network as the underlying classifier. The prototype classifier is inherently resistant to non-characters, and so, can be trained with character and string samples without the need of data augmentation. The learning has two stages: pre-training on character samples with a modified loss function for improving non-character resistance, and weakly supervised learning on both character and string samples for improving recognition performance. Experimental results on the CASIA-OLHWDB and ICDAR2013-Online datasets show that the proposed method can achieve promising recognition performance without training data augmentation.
Content-Aware Urdu Handwriting Generation

The performance of handwriting recognition systems has undergone significant improvement in recent years. However, the accuracy of these systems for multiple cursive scripts, including Arabic and Urdu, is still limited due to the lack of labeled training data. Handwriting generators are a potential solution to this problem. Previous research on Urdu handwriting generation has primarily focused on generating realistic ligatures using Generative Adversarial Networks (GANs) with common adversarial loss but has not addressed the issue of maintaining content and generated image entanglement. This paper aims to address this gap by proposing a content-controlled training approach for Urdu Handwriting Generation with pre-trained recognizer loss. Our generation model is trained on a diverse set of printed ligatures and then fine-tuned with transfer learning on handwritten images. In this paper, a new metric for evaluating the performance of handwriting generation systems is also suggested, which is specifically tailored to the context of handwriting generation tasks. To our knowledge, this is the first Urdu handwriting generation system that is capable of generating content-controlled images.
Private Inference for Deep Neural Networks: A Secure, Adaptive, and Efficient Realization

The advances in deep neural networks (DNNs) have driven many companies to offer their carefully-trained DNNs as inference services for clients’ private data. The privacy concerns have increasingly motivated the need for private inference (PI), where DNN inferences are performed directly on encrypted data without revealing the client’s private inputs to the server or revealing the server’s proprietary DNN weights to the client. However, existing cryptographic protocols for PI suffer from impractically high latency, stemming mostly from non-linear operators like ReLU activations. In this paper, we propose PAPI, a Practical and Adaptive Private Inference framework. First, we develop an accuracy-adaptive neural architecture search (NAS) approach to generate DNN models tailored for high-efficiency ciphertext computation. Specifically, our NAS automatically generates the DNNs with fewer ReLUs while keeping the accuracy above a user-defined target. Second, we propose secure online/offline protocols for ReLU activation and its approximation variants (i.e., polynomial activations), which purely rely on the lightweight secret sharing techniques in the online execution and can well cope with our optimized DNNs in the ciphertext domain. Experimental results show that PAPI reduces online inference latency on the CIFAR-10/100 and ImageNet datasets by 2.7× ∼7.8× over the state-of-the-art.
Design of medical equipment integrated management system based on Internet of Things

In view of the poor effect of integrated management of medical equipment in China, this paper puts forward the design method of integrated management system of medical equipment based on Internet of Things. Optimizes the hardware structure configuration of the system based on Internet of Things technology, improves the system software function and operation process, and constructs a integrated management evaluation system of medical equipment. Finally, it is confirmed by experiment. The integrated management system of medical equipment based on Internet of Things has high practicability in the process of practical application. It can effectively manage massive medical equipment in real time, ensure the accuracy of information management and fully meet the research requirements.
Comparing Security in eBPF and WebAssembly

This paper examines the security of eBPF and WebAssembly (Wasm), two technologies that have gained widespread adoption in recent years, despite being designed for very different use cases and environments. While eBPF is a technology primarily used within operating system kernels such as Linux, Wasm is a binary instruction format designed for a stack-based virtual machine with use cases extending beyond the web. Recognizing the growth and expanding ambitions of eBPF, Wasm may provide instructive insights, given its design around securely executing arbitrary untrusted programs in complex and hostile environments such as web browsers and clouds. We analyze the security goals, community evolution, memory models, and execution models of both technologies, and conduct a comparative security assessment, exploring memory safety, control flow integrity, API access, and side-channels. Our results show that eBPF has a history of focusing on performance first and security second, while Wasm puts more emphasis on security at the cost of some runtime overheads. Considering language-based restrictions for eBPF and a security model for API access are fruitful directions for future work.
Certificateless public auditing scheme with sensitive information hiding for data sharing in cloud storage

In the cloud storage environment, users can store their data on cloud servers to save local storage resources and also share data with other users through cloud servers. Nevertheless, the outsourced data in cloud is vulnerable to tampering or loss due to software or hardware failures, hacker attacks, and other unforeseen issues. Public auditing technique have been developed to safeguard the integrity of the outsourced data. In addition, outsourced data typically contains users’ sensitive information like their names and ages, which are likely to be unavoidably exposed to cloud servers and other users. To tackle the aforementioned issues, a certificateless public auditing scheme with sensitive information hiding (CPAS-SIH) for data sharing is presented in this paper, which introduces a third-party sanitizer to sanitize the data blocks encompassing sensitive information in outsourced data and converts their tags into the valid ones for sanitized data blocks. Meanwhile, an extended double linked list information table is utilized to enable data dynamic operations, including the data blocks modification, insertion, as well as deletion. The security analysis demonstrates that CPAS-SIH satisfies unforgeability, auditing soundness, immutability, and data privacy preservation. The performance evaluation indicates that CPAS-SIH is efficient and practical.
SSTM: Spatiotemporal recurrent transformers for multi-frame optical flow estimation

Inaccurate optical flow estimates in and near occluded regions, and out-of-boundary regions are two of the current significant limitations of optical flow estimation algorithms. Recent state-of-the-art optical flow estimation algorithms are two-frame based methods where optical flow is estimated sequentially for each consecutive image pair in a sequence. While this approach gives good flow estimates, it fails to generalize optical flows in occluded regions mainly due to limited local evidence regarding moving elements in a scene. In this work, we propose a learning-based multi-frame optical flow estimation method that estimates two or more consecutive optical flows in parallel from multi-frame image sequences. Our underlying hypothesis is that by understanding temporal scene dynamics from longer sequences with more than two frames, we can characterize pixel-wise dependencies in a larger spatiotemporal domain, generalize complex motion patterns and thereby improve the accuracy of optical flow estimates in occluded regions. We present learning-based spatiotemporal recurrent transformers for multi-frame based optical flow estimation (SSTMs). Our method utilizes 3D Convolutional Gated Recurrent Units (3D-ConvGRUs) and spatiotemporal transformers to learn recurrent space–time motion dynamics and global dependencies in the scene and provide a generalized optical flow estimation. When compared with recent state-of-the-art two-frame and multi-frame methods on real world and synthetic datasets, performance of the SSTMs were significantly higher in occluded and out-of-boundary regions. Among all published state-of-the-art multi-frame methods, SSTM achieved state-of the-art results on the Sintel Final and KITTI2015 benchmark datasets. Software code, data and instructions: https://github.com/Computational-Ocularscience/SSTM.
Machine learning techniques for classifying dangerous asteroids

There is an infinite number of objects in outer space, and these objects and asteroids might be harmful. Hence, it is wise to know what is surrounding us and what can harm us amongst those. Therefore, in this article, with the hyperparameters tuning of Extra Tree, Random Forest, Light Gradient Boosting Machine, Gradient Boosting, and Ada Boost, the hazards of asteroids around the Earth were classified, and the results of ROC Curves for these algorithms were compared.•Reviewing the list of NASA-certified asteroids classified as the nearest Earth object•Investigating the risk of asteroids with the help of Extra Tree, Random Forest, Light Gradient Boosting Machine, Gradient Boosting, and Ada Boost•Comparing the performance of machine learning algorithms in the classification of high-risk asteroids.
Genetic algorithms-based optimal site selection of solar PV in the north of Afghanistan

Formulae display:?Mathematical formulae have been encoded as MathML and are displayed in this HTML version using MathJax in order to improve their display. Uncheck the box to turn MathJax off. This feature requires Javascript. Click on a formula to zoom.
Language agents reduce the risk of existential catastrophe

Recent advances in natural-language processing have given rise to a new kind of AI architecture: the language agent. By repeatedly calling an LLM to perform a variety of cognitive tasks, language agents are able to function autonomously to pursue goals specified in natural language and stored in a human-readable format. Because of their architecture, language agents exhibit behavior that is predictable according to the laws of folk psychology: they function as though they have desires and beliefs, and then make and update plans to pursue their desires given their beliefs. We argue that the rise of language agents significantly reduces the probability of an existential catastrophe due to loss of control over an AGI. This is because the probability of such an existential catastrophe is proportional to the difficulty of aligning AGI systems, and language agents significantly reduce that difficulty. In particular, language agents help to resolve three important issues related to aligning AIs: reward misspecification, goal misgeneralization, and uninterpretability.