OpenRaiser/Intern-Atlas huggingface dataset:
The Who in XAI: How AI Background Shapes Perceptions of AI Explanations
Venue: arXiv cs.HC 2024
Year: 2024

Explainability of AI systems is critical for users to take informed actions. Understanding "who" opens the black-box of AI is just as important as opening it. We conduct a mixed-methods study of how two different groups--people with and without AI background--perceive different types of AI explanations. Quantitatively, we share user perceptions along five dimensions. Qualitatively, we describe how AI background can influence interpretations, elucidating the differences through lenses of appropriation and cognitive heuristics. We find that (1) both groups showed unwarranted faith in numbers for different reasons and (2) each group found value in different explanations beyond their intended design. Carrying critical implications for the field of XAI, our findings showcase how AI generated explanations can have negative consequences despite best intentions and how that could lead to harmful manipulation of trust. We propose design interventions to mitigate them.
Speech2AffectiveGestures: Synthesizing Co-Speech Gestures with Generative Adversarial Affective Expression Learning
Venue: arXiv cs.MM 2024
Year: 2024

We present a generative adversarial network to synthesize 3D pose sequences of co-speech upper-body gestures with appropriate affective expressions. Our network consists of two components: a generator to synthesize gestures from a joint embedding space of features encoded from the input speech and the seed poses, and a discriminator to distinguish between the synthesized pose sequences and real 3D pose sequences. We leverage the Mel-frequency cepstral coefficients and the text transcript computed from the input speech in separate encoders in our generator to learn the desired sentiments and the associated affective cues. We design an affective encoder using multi-scale spatial-temporal graph convolutions to transform 3D pose sequences into latent, pose-based affective features. We use our affective encoder in both our generator, where it learns affective features from the seed poses to guide the gesture synthesis, and our discriminator, where it enforces the synthesized gestures to contain the appropriate affective expressions. We perform extensive evaluations on two benchmark datasets for gesture synthesis from the speech, the TED Gesture Dataset and the GENEA Challenge 2020 Dataset. Compared to the best baselines, we improve the mean absolute joint error by 10--33%, the mean acceleration difference by 8--58%, and the Fréchet Gesture Distance by 21--34%. We also conduct a user study and observe that compared to the best current baselines, around 15.28% of participants indicated our synthesized gestures appear more plausible, and around 16.32% of participants felt the gestures had more appropriate affective expressions aligned with the speech.
Measuring User Experience Inclusivity in Human-AI Interaction via Five User Problem-Solving Styles
Venue: arXiv cs.HC 2024
Year: 2024

Motivations: Recent research has emerged on generally how to improve AI product user experiences, but relatively little is known about an AI product's inclusivity. For example, what kinds of users does it support well, and who does it leave out? And what changes in the product would make it more inclusive? Objectives: Our overall objective is to help fill this gap, investigating what kinds of diverse users an AI product leaves out, and how to act upon that knowledge. To bring actionability to our findings, we focus on users' diversity of problem-solving attributes. Thus, our specific objectives were: (1) to reveal whether participants with diverse problem-solving styles were left behind in a set of AI products; and (2) to relate participants' problem-solving diversity to their demographic diversity, specifically, gender and age. Methods: We performed 18 experiments, discarding two that failed manipulation checks. Each experiment was a 2x2 factorial experiment with online participants. Each experiment compared two AI products: one deliberately violating an HAI guideline and the other applying the guideline. For our first objective, we analyzed how much each AI product gained/lost inclusivity compared to its counterpart, where inclusivity was supportiveness to participants with particular problem-solving styles. For our second objective, we analyzed how participants' problem-solving styles aligned with their demographics, namely their genders and ages. Results & Implications: Participants' diverse problem-solving styles revealed six types of inclusivity results: (1) the AI products that followed an HAI guideline were almost always more inclusive across diversity of problem-solving styles than the products that did not follow that guideline-but the "who" that got most of the inclusivity varied widely by guideline and by problem-solving style...
Self-supervised Audiovisual Representation Learning for Remote Sensing Data
Venue: arXiv cs.CV 2024
Year: 2024

Many current deep learning approaches make extensive use of backbone networks pre-trained on large datasets like ImageNet, which are then fine-tuned to perform a certain task. In remote sensing, the lack of comparable large annotated datasets and the wide diversity of sensing platforms impedes similar developments. In order to contribute towards the availability of pre-trained backbone networks in remote sensing, we devise a self-supervised approach for pre-training deep neural networks. By exploiting the correspondence between geo-tagged audio recordings and remote sensing imagery, this is done in a completely label-free manner, eliminating the need for laborious manual annotation. For this purpose, we introduce the SoundingEarth dataset, which consists of co-located aerial imagery and audio samples all around the world. Using this dataset, we then pre-train ResNet models to map samples from both modalities into a common embedding space, which encourages the models to understand key properties of a scene that influence both visual and auditory appearance. To validate the usefulness of the proposed approach, we evaluate the transfer learning performance of pre-trained weights obtained against weights obtained through other means. By fine-tuning the models on a number of commonly used remote sensing datasets, we show that our approach outperforms existing pre-training strategies for remote sensing imagery. The dataset, code and pre-trained model weights will be available at https://github.com/khdlr/SoundingEarth.
Modified Double DQN: addressing stability
Venue: arXiv cs.AI 2024
Year: 2024

Inspired by Double Q-learning algorithm, the Double-DQN (DDQN) algorithm was originally proposed in order to address the overestimation issue in the original DQN algorithm. The DDQN has successfully shown both theoretically and empirically the importance of decoupling in terms of action evaluation and selection in computation of target values; although, all the benefits were acquired with only a simple adaption to DQN algorithm, minimal possible change as it was mentioned by the authors. Nevertheless, there seems a roll-back in the proposed algorithm of DDQN since the parameters of policy network are emerged again in the target value function which were initially withdrawn by DQN with the hope of tackling the serious issue of moving targets and the instability caused by it (i.e., by moving targets) in the process of learning. Therefore, in this paper three modifications to the DDQN algorithm are proposed with the hope of maintaining the performance in the terms of both stability and overestimation. These modifications are focused on the logic of decoupling the best action selection and evaluation in the target value function and the logic of tackling the moving targets issue. Each of these modifications have their own pros and cons compared to the others. The mentioned pros and cons mainly refer to the execution time required for the corresponding algorithm and the stability provided by the corresponding algorithm. Also, in terms of overestimation, none of the modifications seem to underperform compared to the original DDQN if not outperform it. With the intention of evaluating the efficacy of the proposed modifications, multiple empirical experiments along with theoretical experiments were conducted. The results obtained are represented and discussed in this article.
Complexity of Inexact Proximal Point Algorithm for minimizing convex functions with Holderian Growth
Venue: arXiv cs.LG 2024
Year: 2024

Several decades ago the Proximal Point Algorithm (PPA) started to gain a long-lasting attraction for both abstract operator theory and numerical optimization communities. Even in modern applications, researchers still use proximal minimization theory to design scalable algorithms that overcome nonsmoothness. Remarkable works as \cite{Fer:91,Ber:82constrained,Ber:89parallel,Tom:11} established tight relations between the convergence behaviour of PPA and the regularity of the objective function. In this manuscript we derive nonasymptotic iteration complexity of exact and inexact PPA to minimize convex functions under $γ-$Holderian growth: $\BigO{\log(1/ε)}$ (for $γ\in [1,2]$) and $\BigO{1/ε^{γ- 2}}$ (for $γ> 2$). In particular, we recover well-known results on PPA: finite convergence for sharp minima and linear convergence for quadratic growth, even under presence of deterministic noise. Moreover, when a simple Proximal Subgradient Method is recurrently called as an inner routine for computing each IPPA iterate, novel computational complexity bounds are obtained for Restarting Inexact PPA. Our numerical tests show improvements over existing restarting versions of the Subgradient Method.
Approximation and generalization properties of the random projection classification method
Venue: arXiv cs.LG 2024
Year: 2024

The generalization gap of a classifier is related to the complexity of the set of functions among which the classifier is chosen. We study a family of low-complexity classifiers consisting of thresholding a random one-dimensional feature. The feature is obtained by projecting the data on a random line after embedding it into a higher-dimensional space parametrized by monomials of order up to k. More specifically, the extended data is projected n-times and the best classifier among those n, based on its performance on training data, is chosen. We show that this type of classifier is extremely flexible as, given full knowledge of the class conditional densities, under mild conditions, the error of these classifiers would converge to the optimal (Bayes) error as k and n go to infinity. We also bound the generalization gap of the random classifiers. In general, these bounds are better than those for any classifier with VC dimension greater than O(ln n). In particular, the bounds imply that, unless the number of projections n is extremely large, the generalization gap of the random projection approach is significantly smaller than that of a linear classifier in the extended space. Thus, for certain classification problems (e.g., those with a large Rashomon ratio), there is a potntially large gain in generalization properties by selecting parameters at random, rather than selecting the best one amongst the class.
The Fine-Grained Complexity of Episode Matching
Venue: arXiv cs.DS 2024
Year: 2024

Given two strings $S$ and $P$, the Episode Matching problem is to find the shortest substring of $S$ that contains $P$ as a subsequence. The best known upper bound for this problem is $\tilde O(nm)$ by Das et al. (1997) , where $n,m$ are the lengths of $S$ and $P$, respectively. Although the problem is well studied and has many applications in data mining, this bound has never been improved. In this paper we show why this is the case by proving that no $O((nm)^{1-ε})$ algorithm (even for binary strings) exists, unless the Strong Exponential Time Hypothesis (SETH) is false. We then consider the indexing version of the problem, where $S$ is preprocessed into a data structure for answering episode matching queries $P$. We show that for any $τ$, there is a data structure using $O(n+\left(\frac{n}τ\right)^k)$ space that answers episode matching queries for any $P$ of length $k$ in $O(k\cdot τ\cdot \log \log n )$ time. We complement this upper bound with an almost matching lower bound, showing that any data structure that answers episode matching queries for patterns of length $k$ in time $O(n^δ)$, must use $Ω(n^{k-kδ-o(1)})$ space, unless the Strong $k$-Set Disjointness Conjecture is false. Finally, for the special case of $k=2$, we present a faster construction of the data structure using fast min-plus multiplication of bounded integer matrices.
INVIGORATE: Interactive Visual Grounding and Grasping in Clutter
Venue: arXiv cs.RO 2024
Year: 2024

This paper presents INVIGORATE, a robot system that interacts with human through natural language and grasps a specified object in clutter. The objects may occlude, obstruct, or even stack on top of one another. INVIGORATE embodies several challenges: (i) infer the target object among other occluding objects, from input language expressions and RGB images, (ii) infer object blocking relationships (OBRs) from the images, and (iii) synthesize a multi-step plan to ask questions that disambiguate the target object and to grasp it successfully. We train separate neural networks for object detection, for visual grounding, for question generation, and for OBR detection and grasping. They allow for unrestricted object categories and language expressions, subject to the training datasets. However, errors in visual perception and ambiguity in human languages are inevitable and negatively impact the robot's performance. To overcome these uncertainties, we build a partially observable Markov decision process (POMDP) that integrates the learned neural network modules. Through approximate POMDP planning, the robot tracks the history of observations and asks disambiguation questions in order to achieve a near-optimal sequence of actions that identify and grasp the target object. INVIGORATE combines the benefits of model-based POMDP planning and data-driven deep learning. Preliminary experiments with INVIGORATE on a Fetch robot show significant benefits of this integrated approach to object grasping in clutter with natural language interactions. A demonstration video is available at https://youtu.be/zYakh80SGcU.
Subgoal Search For Complex Reasoning Tasks
Venue: arXiv cs.AI 2024
Year: 2024

Humans excel in solving complex reasoning tasks through a mental process of moving from one idea to a related one. Inspired by this, we propose Subgoal Search (kSubS) method. Its key component is a learned subgoal generator that produces a diversity of subgoals that are both achievable and closer to the solution. Using subgoals reduces the search space and induces a high-level search graph suitable for efficient planning. In this paper, we implement kSubS using a transformer-based subgoal module coupled with the classical best-first search framework. We show that a simple approach of generating $k$-th step ahead subgoals is surprisingly efficient on three challenging domains: two popular puzzle games, Sokoban and the Rubik's Cube, and an inequality proving benchmark INT. kSubS achieves strong results including state-of-the-art on INT within a modest computational budget.
Multilingual Multi-Aspect Explainability Analyses on Machine Reading Comprehension Models
Venue: arXiv cs.CL 2024
Year: 2024

Achieving human-level performance on some of the Machine Reading Comprehension (MRC) datasets is no longer challenging with the help of powerful Pre-trained Language Models (PLMs). However, the internal mechanism of these artifacts remains unclear, placing an obstacle for further understanding these models. This paper focuses on conducting a series of analytical experiments to examine the relations between the multi-head self-attention and the final MRC system performance, revealing the potential explainability in PLM-based MRC models. To ensure the robustness of the analyses, we perform our experiments in a multilingual way on top of various PLMs. We discover that passage-to-question and passage understanding attentions are the most important ones in the question answering process, showing strong correlations to the final performance than other parts. Through comprehensive visualizations and case studies, we also observe several general findings on the attention maps, which can be helpful to understand how these models solve the questions.
Continual learning under domain transfer with sparse synaptic bursting
Venue: arXiv cs.LG 2024
Year: 2024

Existing machines are functionally specific tools that were made for easy prediction and control. Tomorrow's machines may be closer to biological systems in their mutability, resilience, and autonomy. But first they must be capable of learning and retaining new information without being exposed to it arbitrarily often. Past efforts to engineer such systems have sought to build or regulate artificial neural networks using disjoint sets of weights that are uniquely sensitive to specific tasks or inputs. This has not yet enabled continual learning over long sequences of previously unseen data without corrupting existing knowledge: a problem known as catastrophic forgetting. In this paper, we introduce a system that can learn sequentially over previously unseen datasets (ImageNet, CIFAR-100) with little forgetting over time. This is done by controlling the activity of weights in a convolutional neural network on the basis of inputs using top-down regulation generated by a second feed-forward neural network. We find that our method learns continually under domain transfer with sparse bursts of activity in weights that are recycled across tasks, rather than by maintaining task-specific modules. Sparse synaptic bursting is found to balance activity and suppression such that new functions can be learned without corrupting extant knowledge, thus mirroring the balance of order and disorder in systems at the edge of chaos. This behavior emerges during a prior pre-training (or 'meta-learning') phase in which regulated synapses are selectively disinhibited, or grown, from an initial state of uniform suppression through prediction error minimization.
Toward an Actionable Socioeconomic-Aware HCI
Venue: arXiv cs.HC 2024
Year: 2024

Although inequities for individuals in different socioeconomic situations are starting to capture widespread attention, less attention has been given to the socioeconomic inequities that saturate socioeconomic-diverse individuals' user experiences. To enable HCI practitioners to attend to such inequities and avoid unwittingly introducing them, in this paper we consider a wide body of research relevant to how an individual's socioeconomic status (SES) can affect their user experiences with technology. We synthesize this foundational research to produce a core set of 6 evidence-based SES "facets" (attribute types and value ranges) that directly relate to user experiences for individuals in different SES strata. We then harness these SES facets to produce actionable paths forward -- including a new structured method we call SocioeconomicMag -- by which HCI researchers and practitioners can bring new socioeconomic-aware practices into their everyday HCI work.
Matrix hypercontractivity, streaming algorithms and LDCs: the large alphabet case
Venue: arXiv cs.CC 2024
Year: 2024

We prove a hypercontractive inequality for matrix-valued functions defined over large alphabets. In order to do so, we prove a generalization of the powerful $2$-uniform convexity inequality for trace norms of Ball, Carlen, Lieb (Inventiones Mathematicae'94). Using our hypercontractive~inequality, we present upper and lower bounds for the communication complexity of the Hidden Hypermatching problem defined over large alphabets. We then consider streaming algorithms for approximating the value of Unique Games on a hypergraph with $t$-size hyperedges. By using our communication lower bound, we show that every streaming algorithm in the adversarial model achieving an $(r-\varepsilon)$-approximation of this value requires $Ω(n^{1-2/t})$ quantum space, where $r$ is the alphabet size. We next present a lower bound for locally decodable codes (LDC) $\mathbb{Z}_r^n\to \mathbb{Z}_r^N$ over large alphabets with recoverability probability at least $1/r + \varepsilon$. Using hypercontractivity, we give an exponential lower bound $N = 2^{Ω(\varepsilon^4 n/r^4)}$ for $2$-query (possibly non-linear) LDCs over $\mathbb{Z}_r$ and using the non-commutative Khintchine inequality we prove an improved lower bound of $N = 2^{Ω(\varepsilon^2 n/r^2)}$.
The decidability of the genus of regular languages and directed emulators
Venue: arXiv cs.FL 2024
Year: 2024

The article continues our study of the genus of a regular language $L$, defined as the minimal genus among all genera of all finite deterministic automata recognizing $L$. Here we define and study two closely related tools on a directed graph: directed emulators and automatic relations. A directed emulator morphism essentially encapsulates at the graph-theoretic level an epimorphism onto the minimal deterministic automaton. An automatic relation is the graph-theoretic version of the Myhill-Nerode relation. We show that an automatic relation determines a directed emulator morphism and respectively, a directed emulator morphism determines an automatic relation up to isomorphism. Consider the set $S$ of all directed emulators of the underlying directed graph of the minimal deterministic automaton for $L$. We prove that the genus of $L$ is $\underset{G \in S}{\min}\ g(G)$. We also consider the more restrictive notion of directed cover and prove that the genus of $L$ is reached in the class of directed covers of the underlying directed graph of the minimal deterministic automaton for $L$. This stands in sharp contrast to undirected emulators and undirected covers which we also consider. Finally we prove that if the problem of determining the minimal genus of a directed emulator of a directed graph has a solution then the problem of determining the minimal genus of an undirected emulator of an undirected graph has a solution.
A Note on Knowledge Distillation Loss Function for Object Classification
Venue: arXiv cs.LG 2024
Year: 2024

This research note provides a quick introduction to the knowledge distillation loss function used in object classification. In particular, we discuss its connection to a previously proposed logits matching loss function. We further treat knowledge distillation as a specific form of output regularization and demonstrate its connection to label smoothing and entropy-based regularization.
A Double-Linked Blockchain Approach Based on Proof-of-Refundable-Tax Consensus Algorithm
Venue: arXiv cs.CR 2024
Year: 2024

In this paper we propose a double-linked blockchain data structure that greatly improves blockchain performance and guarantees single chain with no forks. Additionally, with the proposed proof-of-refundable-tax (PoRT) consensus algorithm, our approach can construct highly reliable, efficient, fair and stable blockchain operations. The PoRT algorithm adopts a verifiable random function instead of mining to select future block maintainers with the probability proportional to each participant's personal refundable tax. The individual refundable tax serves as an index of the activeness of participation and hence PoRT can effectively prevent Sybil attacks. Also, with the block-completion reward deducted from each maintainer's refundable tax, our blockchain system maintains a stable wealth distribution and avoids the "rich become richer" problem. We have implemented the approach and tested with very promising results.