Reviewerly/exHarmony huggingface dataset:
FedVLN: Privacy-Preserving Federated Vision-and-Language Navigation

Data privacy is a central problem for embodied agents that can perceive the environment, communicate with humans, and act in the real world. While helping humans complete tasks, the agent may observe and process sensitive information of users, such as house environments, human activities, etc. In this work, we introduce privacy-preserving embodied agent learning for the task of Vision-and-Language Navigation (VLN), where an embodied agent navigates house environments by following natural language instructions. We view each house environment as a local client, which shares nothing other than local updates with the cloud server and other clients, and propose a novel Federated Vision-and-Language Navigation (FedVLN) framework to protect data privacy during both training and pre-exploration. Particularly, we propose a decentralized federated training strategy to limit the data of each client to its local model training and a federated pre-exploration method to do partial model aggregation to improve model generalizability to unseen environments. Extensive results on R2R and RxR datasets show that, decentralized federated training achieve comparable results with centralized training while protecting seen environment privacy, and federated pre-exploration significantly outperforms centralized pre-exploration while preserving unseen environment privacy. Code is available at https://github.com/eric-ai-lab/FedVLN .
Attributable Visual Similarity Learning

This paper proposes an attributable visual similarity learning (AVSL) framework for a more accurate and ex-plainable similarity measure between images. Most existing similarity learning methods exacerbate the unexplain-ability by mapping each sample to a single point in the em-bedding space with a distance metric (e.g., Mahalanobis distance, Euclidean distance). Motivated by the human se-mantic similarity cognition, we propose a generalized simi-larity learning paradigm to represent the similarity between two images with a graph and then infer the overall simi-larity accordingly. Furthermore, we establish a bottom-up similarity construction and top-down similarity inference framework to infer the similarity based on semantic hier-archy consistency. We first identify unreliable higher-level similarity nodes and then correct them using the most co-herent adjacent lower-level similarity nodes, which simulta-neously preserve traces for similarity attribution. Extensive experiments on the CUB-200-2011, Cars196, and Stanford Online Products datasets demonstrate significant improve-ments over existing deep similarity learning methods and verify the interpretability of our framework. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Code: https://github.com/zbr17/AVSL.
Privacy Preservation for Time Series Data in the Electricity Sector

The big data era has raised public concern regarding private information leakage. Therefore, in the electricity sector, many classical privacy preserving mechanisms based on noise injection have been designed and implemented for meter data. However, injected noise of large magnitudes can affect the statistical structure of these data. Therefore, in this study, we identify the inherent randomness embedded in time series data to mitigate this issue. To this end, we study the potential of using this inherent randomness to protect the privacy for both high and low resolution time series data. We propose a privacy preserving mechanism using stochastic differential equation modeling. We theoretically prove the effectiveness of our proposed framework and design several methods to implement our mechanism to aid various data-driven consumer behavior analysis tasks in the electricity sector. The numerical results indicate that our framework can simultaneously maintain the desired level of privacy preservation and value of data in practice.
Medical Visual Question Answering via Conditional Reasoning and Contrastive Learning

Medical visual question answering (Med-VQA) aims to accurately answer a clinical question presented with a medical image. Despite its enormous potential in healthcare services, the development of this technology is still in the initial stage. On the one hand, Med-VQA tasks are highly challenging due to the massive diversity of clinical questions that require different visual reasoning skills for different types of questions. On the other hand, medical images are complex in nature and very different from natural images, while current Med-VQA datasets are small-scale with a few hundred radiology images, making it difficult to train a well-performing visual feature extractor. This paper addresses above two critical issues. We propose a novel conditional reasoning mechanism with a question-conditioned reasoning component and a type-conditioned reasoning strategy to learn effective reasoning skills for different Med-VQA tasks adaptively. Further, we propose to pre-train a visual feature extractor for Med-VQA via contrastive learning on large amounts of unlabeled radiology images. The effectiveness of our proposals is validated by extensive experiments on existing Med-VQA benchmarks, which show significant improvement of our model in prediction accuracy over state-of-the-art methods. The source code and pre-training dataset are provided at https://github.com/Awenbocc/CPCR.
A MIP-Based Approach for Multi-Robot Geometric Task-and-Motion Planning

We address multi-robot geometric task-and-motion planning (MR-GTAMP) problems in synchronous, monotone setups. The goal of the MR-GTAMP problem is to move objects with multiple robots to goal regions in the presence of other movable objects. To perform the tasks successfully and effectively, the robots have to adopt intelligent collaboration strategies, i.e., decide which robot should move which objects to which positions, and perform collaborative actions, such as handovers. To endow robots with these collaboration capabilities, we propose to first collect occlusion and reachability information for each robot as well as information about whether two robots can perform a handover action by calling motion-planning algorithms. We then propose a method that uses the collected information to build a graph structure which captures the precedence of the manipulations of different objects and supports the implementation of a mixed-integer program to guide the search for highly effective collaborative task-and-motion plans. The search process for collaborative task-and-motion plans is based on a Monte-Carlo Tree Search (MCTS) exploration strategy to achieve exploration-exploitation balance. We evaluate our framework in two challenging GTAMP domains and show that it can generate high-quality task-and-motion plans with respect to the planning time, the resulting plan length and the number of objects moved compared to two state-of-the-art baselines.
Dimensional Expansion and Time-Series Data Augmentation Policy for Skeleton-Based Pose Estimation

Human pose estimation has long been researched as a significant topic in computer vision. However, studies via deep learning models are insufficient because of the lack of 2D and 3D skeleton data in various domains. An augmentation technique is applied to solve the problem of data scarcity. Data augmentation techniques can improve the performance of an analytical model by increasing the amount of data. However, the model’s performance is degraded if the augmented results differ significantly from the actual distribution. Therefore, it is necessary to implement an optimized augmentation policy for image datasets. This study proposed a dimensional expansion and time-series data augmentation policy for pose estimation based on skeletons. The proposed method improves the model’s performance by using 3D skeleton data. The 3D skeleton data were preprocessed through an affine transformation. The data were augmented for dimensional expansion from 2D to 3D data. In addition, sampling was applied to the data in consideration of time-series features, and thus, the number of frames per unit time was redefined. Subsequently, part of the information was lost by cutout, and thus, the data size, rather than the data shape, was changed. Through the image and video augmentation policies and cutout expansion, search candidates for 3D time-series data augmentation policies were extracted from the number of cases generated through the combination of 16 skeleton augmentation policies, 11 probabilities, and ten intensities. Finally, 20 candidates were extracted, and the five best-performing policies were applied.
Low Communication Complexity Protocols, Collision Resistant Hash Functions and Secret Key-Agreement Protocols

WeCohen, Shahar P. Naor, Moni study communication complexity in computational settings where bad inputs may exist, but they should be hard to find for any computationally bounded adversary. We define a model where there is a source of public randomness but the inputs are chosen by a computationally bounded adversarial participant after seeing the public randomness. We show that breaking the known communication lower bounds of the private coins model in this setting is closely connected to known cryptographic assumptions. We consider the simultaneous messages model and the interactive communication model and show that for any non trivial predicate (with no redundant rows, such as equality): The other model we study is that of a stateful “free talk”, where participants can communicate freely before the inputs are chosen and may maintain a state, and the communication complexity is measured only afterwards. We show that efficient protocols for equality in this model imply secret key-agreement protocols in a constructive manner. On the other hand, secret key-agreement protocols imply optimal (in terms of error) protocols for equality.
Aspect-level sentiment analysis based on BERT fusion multi-attention

In view of the fact that the existing aspect-level sentiment analysis (ABSA) model cannot effectively distinguish the importance of aspect words and words in the text, and lacks the utilization of the overall interaction between aspect words and text, an aspect-level approach based on BERT combined with multi-attention is proposed. The sentiment analysis model captures the interaction and correlation between aspect words and the overall text sentence through the text and aspect word interactive attention mechanism, thereby improving the accuracy of ABSA. Comparative experiments are carried out on the restaurant and laptop datasets of the Semeval2014 evaluation task. The experimental results show that the model proposed in this paper achieves good classification results in the aspect-level sentiment analysis task for short text reviews. This method provides a new idea for ABSA for review texts.
An empirical investigation on the trade-off between smart contract readability and gas consumption

Blockchain technology is becoming increasingly popular, and smart contracts (i.e., programs that run on top of the blockchain) represent a crucial element of this technology. In particular, smart contracts running on Ethereum (i.e., one of the most popular blockchain platforms) are often developed with Solidity, and their deployment and execution consume gas (i.e., a fee compensating the computing resources required). Smart contract development frequently involves code reuse, but poor readable smart contracts could hinder their reuse. However, writing readable smart contracts is challenging, since practices for improving the readability could also be in contrast with optimization strategies for reducing gas consumption. This paper aims at better understanding (i) the readability aspects for which traditional software and smart contracts differ, and (ii) the specific smart contract readability features exhibiting significant relationships with gas consumption. We leverage a set of metrics that previous research has proven correlated with code readability. In particular, we first compare the values of these metrics obtained for both Solidity smart contracts and traditional software systems (written in Java). Then, we investigate the correlations occurring between these metrics and gas consumption and between each pair of metrics. The results of our study highlight that smart contracts usually exhibit lower readability than traditional software for what concerns the number of parentheses, inline comments, and blank lines used. In addition, we found some readability metrics (such as the average length of identifiers and the average number of keywords) that significantly correlate with gas consumption.
Bayesian Optimization with Clustering and Rollback for CNN Auto Pruning

Pruning is an effective technique for convolutional neural networks (CNNs) model compression, but it is difficult to find the optimal pruning policy due to the large design space. To improve the usability of pruning, many auto pruning methods have been developed. Recently, Bayesian optimization (BO) has been considered to be a competitive algorithm for auto pruning due to its solid theoretical foundation and high sampling efficiency. However, BO suffers from the curse of dimensionality. The performance of BO deteriorates when pruning deep CNNs, since the dimension of the design spaces increase. We propose a novel clustering algorithm that reduces the dimension of the design space to speed up the searching process. Subsequently, a rollback algorithm is proposed to recover the high-dimensional design space so that higher pruning accuracy can be obtained. We validate our proposed method on ResNet, MobileNetV1, and MobileNetV2 models. Experiments show that the proposed method significantly improves the convergence rate of BO when pruning deep CNNs with no increase in running time. The source code is available at https://github.com/fanhanwei/BOCR.
Boosting Event Stream Super-Resolution with a Recurrent Neural Network

Existing methods for event stream super-resolution (SR) either require high-quality and high-resolution frames or underperform for large factor SR. To address these problems, we propose a recurrent neural network for event SR without frames. First, we design a temporal propagation net for incorporating neighboring and long-range event-aware contexts that facilitates event SR. Second, we build a spatiotemporal fusion net for reliably aggregating the spatiotemporal clues of event stream. These two elaborate components are tightly synergized for achieving satisfying event SR results even for 16 $$\times $$ SR. Synthetic and real-world experimental results demonstrate the clear superiority of our method. Furthermore, we evaluate our method on two downstream event-driven applications, i.e., object recognition and video reconstruction, achieving remarkable performance boost over existing methods.
Nearly Optimal Property Preserving Hashing

Property-preserving hashing (PPH) consists of a family of compressing hash functions h such that, for any two inputs x, y, we can correctly identify whether some property P(x, y) holds given only the digests h(x), h(y). In a basic PPH, correctness should hold with overwhelming probability over the choice of h when x, y are worst-case values chosen a-priori and independently of h. In an adversarially robust PPH (RPPH), correctness must hold even when x, y are chosen adversarially and adaptively depending on h. Here, we study (R)PPH for the property that the Hamming distance between x and y is at most t. The notion of (R)PPH was introduced by Boyle, LaVigne and Vaikuntanathan (ITCS '19), and further studied by Fleischhacker, Simkin (Eurocrypt '21) and Fleischhacker, Larsen, Simkin (Eurocrypt '22). In this work, we obtain improved constructions that are conceptually simpler, have nearly optimal parameters, and rely on more general assumptions than prior works. Our results are:
Multilingualism and AI: The Regimentation of Language in the Age of Digital Capitalism

This article examines the effects of commercial digital language technologies on the regimentation of language. Language technologies based on the exploitation of large data sets—from machine translation and automatic text generation to digital voice assistants—are a particular form of human-made sign practice in which traditional language norms interact with the affordances of digital devices and the capitalist interests of those who design them. Such sociotechnological practices construct language hierarchies within the realm of commercially based language technology and can shape both dominant discourses about language in society and epistemologies of language in linguistics. The article focuses on interrelationships between digital language technology and metasemiotic interpretations of language that pertain to multilingualism, language variation, and language prestige. It examines languages as discursive constructs and reviews the role of media technology in shaping language ideology, showing that writing and print have had a crucial impact on modern language concepts. It draws on expert discourse and qualitative interviews with programmers and users and examines ideological effects of digital language technology and the potential epistemological reconfigurations of concepts of language that may emerge as a result.
Multi-Granularity Aggregation Transformer for Joint Video-Audio-Text Representation Learning

Many real-world video-text tasks involve different levels of granularity to represent local and global information with distinct semantics, such as frames and words, clips and sentences, or videos and paragraphs. Most existing multimodal representation learning methods suffer from limitations: (i) Adopting expert systems or manual design to extract more fine-grained local information (such as objects and actions in a video frame) for supervision may lead to information asymmetry since there may no corresponding information among modalities; (ii) Neglecting the hierarchical nature of the data to aggregate different levels of information from different modalities will cause insufficient representations. To alleviate the above issues, in this paper, we propose a Multi-Granularity Aggregation Transformer (MGAT) for joint video-audio-text representation learning. Specifically, for intra-modality, we first design a multi-granularity transformer module to relieve information asymmetry by making full use of local and global information within a single modality from different perspectives. Then, for inter-modality, we develop an attention-guided aggregation module to fuse audio and video information hierarchically. Last, we align the aggregated information with text information at different hierarchical levels via intra- and inter-modality consistency loss and contrastive loss. With the help of more granularity of information, we are able to obtain a well-performed representation model for a variety of tasks, e.g., video-paragraph retrieval and video captioning. Extensive experiments on two challenging benchmarks, i.e., ActivityNet-captions and Youcook2, demonstrate the superiority of our proposed method.
Adaptive heterogeneous particle swarm optimization with comprehensive learning strategy

This paper proposes an adaptive heterogeneous particle swarm optimization with a comprehensive learning strategy for solving single-objective constrained optimization problems. In this algorithm, particles can use an exploration strategy and an exploitation strategy to update their positions. The historical success rates of the two strategies are used to adaptively control the adoption rates of strategies in the next iteration. The search strategy in the canonical particle swarm optimization algorithm is based on elite solutions. As a result, when no particles can discover better solutions for several generations, this algorithm is likely to fall into stagnation. To respond to this challenge, a new strategy is proposed to explore the neighbors of the elite solutions in this study. Finally, a constraint handling method is equipped to the proposed algorithm to make it be able to solve constrained optimization problems. The proposed algorithm is compared with the canonical particle swarm optimization, differential evolution, and several recently proposed algorithms on the benchmark test suite. The Wilcoxon signed-rank test results show that the proposed algorithm is significantly better on most of the benchmark problems compared with the competitors. The proposed algorithm is also applied to solve two real-world mechanical engineering problems. The experimental results show that the proposed algorithm performs consistently well on these problems.
Citational practices as a site of resistance and radical pedagogy

Discursive infrastructures are forms of writing that remain mostly invisible but shape higher-level practices built upon their base. This article argues that citational practices are a form of discursive infrastructure that are bases that shape our work. Most importantly, we argue that the infrastructural base built through citation practices is in a moment of breakdown as increasing amounts of people call for more just citational practices that surface multiply marginalized and underrepresented (MMU) scholar voices. Consequently, this article both theorizes citations as infrastructure while also focusing on a case study of the MMU scholar database to help build a more equitable and socially just disciplinary infrastructure.
TWD: A New Deep E2E Model for Text Watermark/Caption and Scene Text Detection in Video

Text watermark detection in video images is challenging because text watermark characteristics are different from caption and scene texts in the video images. Developing a successful model for detecting text watermark, caption, and scene texts is an open challenge. This study aims at developing a new Deep End-to-End model for Text Watermark Detection (TWD), caption and scene text in video images. To standardize non-uniform contrast, quality, and resolution, we explore the U-Net3+ model for enhancing poor quality text without affecting high-quality text. Similarly, to address the challenges of arbitrary orientation, text shapes and complex background, we explore Stacked Hourglass Encoded Fourier Contour Embedding Network (SFCENet) by feeding the output of the U-Net3+ model as input. Furthermore, the proposed work integrates enhancement and detection models as an end-to-end model for detecting multi-type text in video images. To validate the proposed model, we create our own dataset (named TW-866), which provides video images containing text watermark, caption (subtitles), as well as scene text. The proposed model is also evaluated on standard natural scene text detection datasets, namely, ICDAR 2019 MLT, CTW1500, Total-Text, and DAST1500. The results show that the proposed method outperforms the existing methods. This is the first work on text watermark detection in video images to the best of our knowledge.