OpenRaiser/Intern-Atlas huggingface dataset:
Recognizing People by Body Shape Using Deep Networks of Images and Words
Venue: arXiv cs.CV 2023
Year: 2023

Common and important applications of person identification occur at distances and viewpoints in which the face is not visible or is not sufficiently resolved to be useful. We examine body shape as a biometric across distance and viewpoint variation. We propose an approach that combines standard object classification networks with representations based on linguistic (word-based) descriptions of bodies. Algorithms with and without linguistic training were compared on their ability to identify people from body shape in images captured across a large range of distances/views (close-range, 100m, 200m, 270m, 300m, 370m, 400m, 490m, 500m, 600m, and at elevated pitch in images taken by an unmanned aerial vehicle [UAV]). Accuracy, as measured by identity-match ranking and false accept errors in an open-set test, was surprisingly good. For identity-ranking, linguistic models were more accurate for close-range images, whereas non-linguistic models fared better at intermediary distances. Fusion of the linguistic and non-linguistic embeddings improved performance at all, but the farthest distance. Although the non-linguistic model yielded fewer false accepts at all distances, fusion of the linguistic and non-linguistic models decreased false accepts for all, but the UAV images. We conclude that linguistic and non-linguistic representations of body shape can offer complementary identity information for bodies that can improve identification in applications of interest.
Video ControlNet: Towards Temporally Consistent Synthetic-to-Real Video Translation Using Conditional Image Diffusion Models
Venue: arXiv cs.CV 2023
Year: 2023

In this study, we present an efficient and effective approach for achieving temporally consistent synthetic-to-real video translation in videos of varying lengths. Our method leverages off-the-shelf conditional image diffusion models, allowing us to perform multiple synthetic-to-real image generations in parallel. By utilizing the available optical flow information from the synthetic videos, our approach seamlessly enforces temporal consistency among corresponding pixels across frames. This is achieved through joint noise optimization, effectively minimizing spatial and temporal discrepancies. To the best of our knowledge, our proposed method is the first to accomplish diverse and temporally consistent synthetic-to-real video translation using conditional image diffusion models. Furthermore, our approach does not require any training or fine-tuning of the diffusion models. Extensive experiments conducted on various benchmarks for synthetic-to-real video translation demonstrate the effectiveness of our approach, both quantitatively and qualitatively. Finally, we show that our method outperforms other baseline methods in terms of both temporal consistency and visual quality.
Colored E-Graph: Equality Reasoning with Conditions
Venue: arXiv cs.PL 2023
Year: 2023

E-graphs are a prominent data structure that has been increasing in popularity in recent years due to their expanding range of applications in various formal reasoning tasks. Often, they are used for equality saturation, a process of deriving consequences through repeatedly applying universally quantified equality formulas via term rewriting. They handle equality reasoning over a large spaces of terms, but are severely limited in their handling of case splitting and other types of logical cuts, especially when compared to other reasoning techniques such as sequent calculi and resolution. The main difficulty is when equality reasoning requires multiple inconsistent assumptions to reach a single conclusion. Ad-hoc solutions, such as duplicating the e-graph for each assumption, are available, but they are notably resource-intensive. We introduce a key observation is that each duplicate e-graph (with an added assumption) corresponds to coarsened congruence relation. Based on that, we present an extension to e-graphs, called Colored E-Graphs, as a way to represent all of the coarsened congruence relations in a single structure. A colored e-graph is a memory-efficient equivalent of multiple copies of an e-graph, with a much lower overhead. This is attained by sharing as much as possible between different cases, while carefully tracking which conclusion is true under which assumption. Support for multiple relations can be thought of as adding multiple "color-coded" layers on top of the original e-graph structure, leading to a large degree of sharing. In our implementation, we introduce optimizations to rebuilding and e-matching. We run experiments and demonstrate that our colored e-graphs can support hundreds of assumptions and millions of terms with space requirements that are an order of magnitude lower, and with similar time requirements.
SWiPE: A Dataset for Document-Level Simplification of Wikipedia Pages
Venue: arXiv cs.CL 2023
Year: 2023

Text simplification research has mostly focused on sentence-level simplification, even though many desirable edits - such as adding relevant background information or reordering content - may require document-level context. Prior work has also predominantly framed simplification as a single-step, input-to-output task, only implicitly modeling the fine-grained, span-level edits that elucidate the simplification process. To address both gaps, we introduce the SWiPE dataset, which reconstructs the document-level editing process from English Wikipedia (EW) articles to paired Simple Wikipedia (SEW) articles. In contrast to prior work, SWiPE leverages the entire revision history when pairing pages in order to better identify simplification edits. We work with Wikipedia editors to annotate 5,000 EW-SEW document pairs, labeling more than 40,000 edits with proposed 19 categories. To scale our efforts, we propose several models to automatically label edits, achieving an F-1 score of up to 70.6, indicating that this is a tractable but challenging NLU task. Finally, we categorize the edits produced by several simplification models and find that SWiPE-trained models generate more complex edits while reducing unwanted edits.
Solving Projected Model Counting by Utilizing Treewidth and its Limits
Venue: arXiv cs.CC 2023
Year: 2023

In this paper, we introduce a novel algorithm to solve projected model counting (PMC). PMC asks to count solutions of a Boolean formula with respect to a given set of projection variables, where multiple solutions that are identical when restricted to the projection variables count as only one solution. Inspired by the observation that the so-called "treewidth" is one of the most prominent structural parameters, our algorithm utilizes small treewidth of the primal graph of the input instance. More precisely, it runs in time O(2^2k+4n2) where k is the treewidth and n is the input size of the instance. In other words, we obtain that the problem PMC is fixed-parameter tractable when parameterized by treewidth. Further, we take the exponential time hypothesis (ETH) into consideration and establish lower bounds of bounded treewidth algorithms for PMC, yielding asymptotically tight runtime bounds of our algorithm. While the algorithm above serves as a first theoretical upper bound and although it might be quite appealing for small values of k, unsurprisingly a naive implementation adhering to this runtime bound suffers already from instances of relatively small width. Therefore, we turn our attention to several measures in order to resolve this issue towards exploiting treewidth in practice: We present a technique called nested dynamic programming, where different levels of abstractions of the primal graph are used to (recursively) compute and refine tree decompositions of a given instance. Finally, we provide a nested dynamic programming algorithm and an implementation that relies on database technology for PMC and a prominent special case of PMC, namely model counting (#Sat). Experiments indicate that the advancements are promising, allowing us to solve instances of treewidth upper bounds beyond 200.
Jointly Reparametrized Multi-Layer Adaptation for Efficient and Private Tuning
Venue: arXiv cs.CL 2023
Year: 2023

Efficient finetuning of pretrained language transformers is becoming increasingly prevalent for solving natural language processing tasks. While effective, it can still require a large number of tunable parameters. This can be a drawback for low-resource applications and training with differential-privacy constraints, where excessive noise may be introduced during finetuning. To this end, we propose a novel language transformer finetuning strategy that introduces task-specific parameters in multiple transformer layers. These parameters are derived from fixed random projections of a single trainable vector, enabling finetuning with significantly fewer parameters while maintaining performance. We achieve within 5% of full finetuning performance on GLUE tasks with as few as 4,100 parameters per task, outperforming other parameter-efficient finetuning approaches that use a similar number of per-task parameters. Besides, the random projections can be precomputed at inference, avoiding additional computational latency. All these make our method particularly appealing for low-resource applications. Finally, our method achieves the best or comparable utility compared to several recent finetuning methods when training with the same privacy constraints, underscoring its effectiveness and potential real-world impact.
Enhancing Human Capabilities through Symbiotic Artificial Intelligence with Shared Sensory Experiences
Venue: arXiv cs.HC 2023
Year: 2023

The merging of human intelligence and artificial intelligence has long been a subject of interest in both science fiction and academia. In this paper, we introduce a novel concept in Human-AI interaction called Symbiotic Artificial Intelligence with Shared Sensory Experiences (SAISSE), which aims to establish a mutually beneficial relationship between AI systems and human users through shared sensory experiences. By integrating multiple sensory input channels and processing human experiences, SAISSE fosters a strong human-AI bond, enabling AI systems to learn from and adapt to individual users, providing personalized support, assistance, and enhancement. Furthermore, we discuss the incorporation of memory storage units for long-term growth and development of both the AI system and its human user. As we address user privacy and ethical guidelines for responsible AI-human symbiosis, we also explore potential biases and inequalities in AI-human symbiosis and propose strategies to mitigate these challenges. Our research aims to provide a comprehensive understanding of the SAISSE concept and its potential to effectively support and enhance individual human users through symbiotic AI systems. This position article aims at discussing poteintial AI-human interaction related topics within the scientific community, rather than providing experimental or theoretical results.
A Telecare System for Use in Traditional Persian Medicine
Venue: arXiv cs.HC 2023
Year: 2023

Persian Medicine (PM) uses wrist temperature/humidity and pulse to determine a person's health status and temperament. However, the diagnosis may depend on the physician's interpretation, hindering the combination of PM with modern medical methods. This study proposes a system for measuring pulse signals and temperament detection based on PM. The system uses recorded thermal distribution, a temperament questionnaire, and a customized pulse measurement device. The collected data can be sent to a physician via a telecare system for interpretation and prescription of medications. The system was clinically implemented for patient care, assessed the temperaments of 34 participants, and recorded thermal images of the wrist, back of the hand, and entire face. The study suggests that a customized device for measuring pulse waves and other criteria based on PM can be incorporated into a telemedicine system, reducing the dependency on PM specialists for diagnosis.
Global Layers: Non-IID Tabular Federated Learning
Venue: arXiv cs.LG 2023
Year: 2023

Data heterogeneity between clients remains a key challenge in Federated Learning (FL), particularly in the case of tabular data. This work presents Global Layers (GL), a novel partial model personalization method robust in the presence of joint distribution $P(X,Y)$ shift and mixed input/output spaces $X \times Y$ across clients. To the best of our knowledge, GL is the first method capable of supporting both client-exclusive features and classes. We introduce two new benchmark experiments for tabular FL naturally partitioned from existing real world datasets: i) UCI Covertype split into 4 clients by "wilderness area" feature, and ii) UCI Heart Disease, SAHeart, UCI Heart Failure, each as clients. Empirical results in these experiments in the full-participant setting show that GL achieves better outcomes than Federated Averaging (FedAvg) and local-only training, with some clients even performing better than their centralized baseline.
Low Precision Quantization-aware Training in Spiking Neural Networks with Differentiable Quantization Function
Venue: arXiv cs.NE 2023
Year: 2023

Deep neural networks have been proven to be highly effective tools in various domains, yet their computational and memory costs restrict them from being widely deployed on portable devices. The recent rapid increase of edge computing devices has led to an active search for techniques to address the above-mentioned limitations of machine learning frameworks. The quantization of artificial neural networks (ANNs), which converts the full-precision synaptic weights into low-bit versions, emerged as one of the solutions. At the same time, spiking neural networks (SNNs) have become an attractive alternative to conventional ANNs due to their temporal information processing capability, energy efficiency, and high biological plausibility. Despite being driven by the same motivation, the simultaneous utilization of both concepts has yet to be thoroughly studied. Therefore, this work aims to bridge the gap between recent progress in quantized neural networks and SNNs. It presents an extensive study on the performance of the quantization function, represented as a linear combination of sigmoid functions, exploited in low-bit weight quantization in SNNs. The presented quantization function demonstrates the state-of-the-art performance on four popular benchmarks, CIFAR10-DVS, DVS128 Gesture, N-Caltech101, and N-MNIST, for binary networks (64.05\%, 95.45\%, 68.71\%, and 99.43\% respectively) with small accuracy drops and up to 31$\times$ memory savings, which outperforms existing methods.
A Study on the Impact of Gender, Employment Status and Academic Discipline on Cyber Hygiene: A Case Study of University of Nigeria, Nsukka
Venue: arXiv cs.CY 2023
Year: 2023

The COVID19 pandemic has helped amplify the importance of Cyber Hygiene. As the reliance on the Internet and IT services increased during the pandemic. This in turn has introduced a new wave of criminal activities such as cybercrimes. With the emergent of COVID19 which lead to increase in cyberattacks incidents, the pattern and sophistication, there is an urgent need to carry out an exploratory study to find out users level of cyber-hygiene knowledge and culture based on gender, employment status and academic discipline. Above this, with many organisations providing for dual mode work pattern or remote and in-person as the pandemic subsides, this study remains very relevant and hence the aim to investigate the cyber hygiene knowledge and compliance of university students and employees of the University of Nigeria, Nsukka (UNN). In addition, it attempts to verify the relationship between demographics such as gender, employment status and academic discipline on cyber hygiene culture among students and employees. The sample population is made of employees and students of UNN, where the employees are either academic staff or non-academic staff. The sample size consisted of three hundred and sixteen (316) participants, one hundred and eight-seven (187) of whom were females and one hundred and twenty-nine (129) were males. The results offer some useful insight on cyber hygiene practices at the university.
Examining risks of racial biases in NLP tools for child protective services
Venue: arXiv cs.CL 2023
Year: 2023

Although much literature has established the presence of demographic bias in natural language processing (NLP) models, most work relies on curated bias metrics that may not be reflective of real-world applications. At the same time, practitioners are increasingly using algorithmic tools in high-stakes settings, with particular recent interest in NLP. In this work, we focus on one such setting: child protective services (CPS). CPS workers often write copious free-form text notes about families they are working with, and CPS agencies are actively seeking to deploy NLP models to leverage these data. Given well-established racial bias in this setting, we investigate possible ways deployed NLP is liable to increase racial disparities. We specifically examine word statistics within notes and algorithmic fairness in risk prediction, coreference resolution, and named entity recognition (NER). We document consistent algorithmic unfairness in NER models, possible algorithmic unfairness in coreference resolution models, and little evidence of exacerbated racial bias in risk prediction. While there is existing pronounced criticism of risk prediction, our results expose previously undocumented risks of racial bias in realistic information extraction systems, highlighting potential concerns in deploying them, even though they may appear more benign. Our work serves as a rare realistic examination of NLP algorithmic fairness in a potential deployed setting and a timely investigation of a specific risk associated with deploying NLP in CPS settings.
Best of Both Distortion Worlds
Venue: arXiv cs.GT 2023
Year: 2023

We study the problem of designing voting rules that take as input the ordinal preferences of $n$ agents over a set of $m$ alternatives and output a single alternative, aiming to optimize the overall happiness of the agents. The input to the voting rule is each agent's ranking of the alternatives from most to least preferred, yet the agents have more refined (cardinal) preferences that capture the intensity with which they prefer one alternative over another. To quantify the extent to which voting rules can optimize over the cardinal preferences given access only to the ordinal ones, prior work has used the distortion measure, i.e., the worst-case approximation ratio between a voting rule's performance and the best performance achievable given the cardinal preferences. The work on the distortion of voting rules has been largely divided into two worlds: utilitarian distortion and metric distortion. In the former, the cardinal preferences of the agents correspond to general utilities and the goal is to maximize a normalized social welfare. In the latter, the agents' cardinal preferences correspond to costs given by distances in an underlying metric space and the goal is to minimize the (unnormalized) social cost. Several deterministic and randomized voting rules have been proposed and evaluated for each of these worlds separately, gradually improving the achievable distortion bounds, but none of the known voting rules perform well in both worlds simultaneously. In this work, we prove that one can achieve the best of both worlds by designing new voting rules, that simultaneously achieve near-optimal distortion guarantees in both distortion worlds. We also prove that this positive result does not generalize to the case where the voting rule is provided with the rankings of only the top-$t$ alternatives of each agent, for $t<m$.
An Insider Threat Mitigation Framework Using Attribute Based Access Control
Venue: arXiv cs.CR 2023
Year: 2023

Insider Threat is a significant and potentially dangerous security issue in corporate settings. It is difficult to mitigate because, unlike external threats, insiders have knowledge of an organization's access policies, access hierarchy, access protocols, and access scheduling. Several approaches to reducing insider threat have been proposed in the literature. However, the integration of access control and moving target defense (MTD) for deceiving insiders has not been adequately discussed. In this paper, we combine MTD, deception, and attribute-based access control to make it more difficult and expensive for an insider to gain unauthorized access. We introduce the concept of correlated attributes into ABAC and extend the ABAC model with MTD by generating mutated policy using the correlated attributes for insider threat mitigation. The evaluation results show that the proposed framework can effectively identify correlated attributes and produce adequate mutated policy without affecting the usability of the access control systems.
Lightening-Transformer: A Dynamically-operated Optically-interconnected Photonic Transformer Accelerator
Venue: arXiv cs.ET 2023
Year: 2023

The wide adoption and significant computing resource of attention-based transformers, e.g., Vision Transformers and large language models (LLM), have driven the demand for efficient hardware accelerators. There is a growing interest in exploring photonics as an alternative technology to digital electronics due to its high energy efficiency and ultra-fast processing speed. Photonic accelerators have shown promising results for CNNs, which mainly rely on weight-static linear operations. However, they encounter issues when efficiently supporting Transformer architectures, questioning the applicability of photonics to advanced ML tasks. The primary hurdle lies in their inefficiency in handling unique workloads in Transformers, i.e., dynamic and full-range tensor multiplication. In this work, we propose Lightening-Transformer, the first light-empowered, high-performance, and energy-efficient photonic Transformer accelerator. To overcome prior designs' fundamental limitations, we introduce a novel dynamically-operated photonic tensor core, DPTC, a crossbar array of interference-based optical vector dot-product engines supporting highly parallel, dynamic, and full-range matrix multiplication. Furthermore, we design a dedicated accelerator that integrates our novel photonic computing cores with photonic interconnects for inter-core data broadcast, fully unleashing the power of optics. Comprehensive evaluations show that ours achieves >2.6x energy and >12x latency reductions compared to prior photonic accelerators and delivers the lowest energy cost and 2 to 3 orders of magnitude lower energy-delay product compared to electronic Transformer accelerators, all while maintaining digital-comparable accuracy. Our work highlights the immense potential of photonics for advanced ML workloads, such as Transformer-backboned LLM. Our work is available at https://github.com/zhuhanqing/Lightening-Transformer.
Automatic Illumination Spectrum Recovery
Venue: arXiv cs.CV 2023
Year: 2023

We develop a deep learning network to estimate the illumination spectrum of hyperspectral images under various lighting conditions. To this end, a dataset, IllumNet, was created. Images were captured using a Specim IQ camera under various illumination conditions, both indoor and outdoor. Outdoor images were captured in sunny, overcast, and shady conditions and at different times of the day. For indoor images, halogen and LED light sources were used, as well as mixed light sources, mainly halogen or LED and fluorescent. The ResNet18 network was employed in this study, but with the 2D kernel changed to a 3D kernel to suit the spectral nature of the data. As well as fitting the actual illumination spectrum well, the predicted illumination spectrum should also be smooth, and this is achieved by the cubic smoothing spline error cost function. Experimental results indicate that the trained model can infer an accurate estimate of the illumination spectrum.