Reviewerly/exHarmony huggingface dataset:
Efficient Multi-Task Scene Analysis with RGB-D Transformers

Scene analysis is essential for enabling autonomous systems, such as mobile robots, to operate in real-world environments. However, obtaining a comprehensive understanding of the scene requires solving multiple tasks, such as panoptic segmentation, instance orientation estimation, and scene classification. Solving these tasks given limited computing and battery capabilities on mobile platforms is challenging. To address this challenge, we introduce an efficient multi-task scene analysis approach, called EMSAFormer, that uses an RGB-D Transformer-based encoder to simultaneously perform the aforementioned tasks. Our approach builds upon the previously published EMSANet. However, we show that the dual CNN-based encoder of EMSANet can be replaced with a single Transformer-based encoder. To achieve this, we investigate how information from both RGB and depth data can be effectively incorporated in a single encoder. To accelerate inference on robotic hardware, we provide a custom NVIDIA TensorRT extension enabling highly optimization for our EMSAFormer approach. Through extensive experiments on the commonly used indoor datasets NYUv2, SUNRGB-D, and ScanNet, we show that our approach achieves state-of-the-art performance while still enabling inference with up to 39.1 FPS on an NVIDIA Jetson AGX Orin 32 GB.
A Multi-task Learning Model for Gold-two-mention Co-reference Resolution

The task of resolving repeated objects in natural languages is known as co-reference resolution. It is an important part of modern natural language processing and semantic cognition as these implicit relationships are particularly difficult in natural language understanding in downstream tasks. Mention identification and mention linking are the two sub-tasks in the general co-reference resolution research community. Gold-two-mention style co-reference resolution is a special type of co-reference resolution that focuses on linking the ambiguous pronoun to one of the two candidate antecedents. In this paper, we proposed a joint learning model that learns mention identification and mention linking tasks together, because we find that the learning of mention identification can provide supportive dependent information for the learning of mention linking. As far as we know, we propose the first model that introduces a multi-task learning framework to the gold-two-mention co-reference resolution task. We find that our proposed model outperforms state-of-the-art baselines and a single-task learning model on three gold-two-mention co-reference resolution datasets. By comparing the errors made by either the single-task learning model or the multi-task learning model, our error analysis also yields interesting findings about in which way our multi-task learning model makes fewer resolution errors.
Cross-Modal Content Inference and Feature Enrichment for Cold-Start Recommendation

Multimedia recommendation aims to fuse the multi-modal information of items for feature enrichment to improve the recommendation performance. However, existing methods typically introduce multi-modal information based on collaborative information to improve the overall recommendation precision, while failing to explore its cold-start recommendation performance. Meanwhile, these above methods are only applicable when such multi-modal data is available. To address this problem, this paper proposes a recommendation framework, named Cross-modal Content Inference and Feature Enrichment Recommendation (CIERec), which exploits the multi-modal information to improve its cold-start recommendation performance. Specifically, CIERec first introduces image annotation as the privileged information to help guide the mapping of unified features from the visual space to the semantic space in the training phase. And then CIERec enriches the content representation with the fusion of collaborative, visual, and cross-modal inferred representations, so as to improve its cold-start recommendation performance. Experimental results on two real-world datasets show that the content representations learned by CIERec are able to achieve superior cold-start recommendation performance over existing visually-aware recommendation algorithms. More importantly, CIERec can consistently achieve significant improvements with different conventional visually-aware backbones, which verifies its universality and effectiveness.
Toolglass and Magic Lenses: The See-Through Interface

ToolglassTM widgets are new user interface tools that can appear, as though on a transparent sheet of glass, between an application and a traditional cursor. They can be positioned with one hand while the other positions the cursor. The widgets provide a rich and concise vocabulary for operating on application objects. These widgets may incorporate visual filters, called Magic Lens&lt;sup&gt;TM&lt;/sup&gt; filters, that modify the presentation of application objects to reveal hidden information, to enhance data of interest, or to suppress distracting information. Together, these tools form a see-through interface that offers many advantages over traditional controls. They provide a new style of interaction that better exploits the user's everyday skills. They can reduce steps, cursor motion, and errors. Many widgets can be provided in a user interface, by designers and by users, without requiring dedicated screen space. In addition, lenses provide rich context-dependent feedback and the ability to view details and context simultaneously. Our widgets and lenses can be combined to form operation and viewing macros, and can be used over multiple applications.
Light Field Rendering

A number of techniques have been proposed for flying through scenes by redisplaying previously rendered or digitized views. Techniques have also been proposed for interpolating between views by warping input images, using depth information or correspondences between multiple images. In this paper, we describe a simple and robust method for generating new views from arbitrary camera positions without depth information or feature matching, simply by combining and resampling the available images. The key to this technique lies in interpreting the input images as 2D slices of a 4D function - the light field. This function completely characterizes the flow of light through unobstructed space in a static scene with fixed illumination.
BFFNet: a bidirectional feature fusion network for semantic segmentation of remote sensing objects

Purpose High-resolution remote sensing images possess a wealth of semantic information. However, these images often contain objects of different sizes and distributions, which make the semantic segmentation task challenging. In this paper, a bidirectional feature fusion network (BFFNet) is designed to address this challenge, which aims at increasing the accurate recognition of surface objects in order to effectively classify special features. Design/methodology/approach There are two main crucial elements in BFFNet. Firstly, the mean-weighted module (MWM) is used to obtain the key features in the main network. Secondly, the proposed polarization enhanced branch network performs feature extraction simultaneously with the main network to obtain different feature information. The authors then fuse these two features in both directions while applying a cross-entropy loss function to monitor the network training process. Finally, BFFNet is validated on two publicly available datasets, Potsdam and Vaihingen. Findings In this paper, a quantitative analysis method is used to illustrate that the proposed network achieves superior performance of 2–6%, respectively, compared to other mainstream segmentation networks from experimental results on two datasets. Complete ablation experiments are also conducted to demonstrate the effectiveness of the elements in the network. In summary, BFFNet has proven to be effective in achieving accurate identification of small objects and in reducing the effect of shadows on the segmentation process. Originality/value The originality of the paper is the proposal of a BFFNet based on multi-scale and multi-attention strategies to improve the ability to accurately segment high-resolution and complex remote sensing images, especially for small objects and shadow-obscured objects.
Platform information sharing and selling contract strategies with blockchain adoption

The recent rapid development of blockchain technology has drawn considerable attention. To examine the impact of blockchain on platform supply chains, we construct a supply chain consisting of two suppliers and an e-commerce platform. Each supplier may use either a reselling or an agency selling contract; therefore, three scenarios are possible: pure agency selling, pure reselling, or hybrid contracts. Considering blockchain’s role in eliminating transaction costs and achieving demand information transparency, we apply a game theory model to study each player’s pricing strategy and profits when blockchain and selling contract strategies vary. We find that the platform’s optimal selling contract and blockchain strategies strongly depend on commission rates, competition intensity, and transaction costs. Suppliers are willing to join the blockchain to gain demand information and reduce transaction costs. With low transaction costs and high competition intensity, the platform may ally itself with the agency supplier and authorize him to join the blockchain to maximize profits, which may ruin social welfare. From a social welfare perspective, we find that a pure agency contract is optimal, whereas the optimal blockchain strategy varies. Specifically, blockchain adoption may not benefit social welfare even when its operations costs are negligible. We further explore a scenario in which two suppliers can choose their selling contracts to derive insights into suppliers’ contract choices and government subsidy policies. Moreover, we explore the impact of marginal production costs on the optimal strategy for platforms.
Quantum Designated Verifier Signature Scheme with Semi-Trusted Third-Party

Designated verifier signature (DVS) schemes can be applied in many scenes, such as E-voting and E-bidding using DVS attributes to protect the secret of participants. Most traditional DVS schemes are vulnerable to quantum computer attacks. Then, an identity-based semi-trusted third-party quantum DVS (STQDVS) scheme is proposed, which is safe for quantum means of attack. In the new STQDVS scheme, the third participant prepares entangled Bell states and allocates them to the signer and the designated verifier (DV). The signer uses the identity information as her public key and the measurement results of corresponding quantum sequences as her private key to generate the signature. The DV applies the identity information as his public key and the measurement results of corresponding quantum sequences as her private key to verify the signature. Non-transferability and hidden origin are guaranteed by the fact that the DV can generate the same signature as the signatory by means of a simulated signature step. Security analysis has shown that attacks such as forgery, inter-resending, and impersonation are ineffective for the proposed scheme and that even third-party cannot forge signatures. On the other hand, the whole process of the scheme does not require any quantum key distribution protocols, quantum one-way functions, and quantum state exchange tests, which makes the STQDVS scheme relatively simple and more efficient than similar schemes.
Extracting the quantum geometric tensor of an optical Raman lattice by Bloch-state tomography

A complete Bloch state tomography is experimentally implemented to directly extract the quantum geometric tensor (QGT) in a two-dimensional optical Raman lattice. By rotating the measurement basis using a momentum-transferred Raman pulse, the expectation values of three Pauli matrices are measured and thus the QGT can be directly extracted.
Five infinite families of binary cyclic codes and their related codes with good parameters

Cyclic codes are an interesting type of linear codes and have wide applications in communication and storage systems due to their efficient encoding and decoding algorithms. Inspired by the recent work on binary cyclic codes published in Tang and Ding (2022) [14], the objectives of this paper are the construction and analyses of five infinite families of binary cyclic codes with parameters [n,k] and (n−6)/3≤k≤2(n+6)/3. Three of the five families of binary cyclic codes and their duals have a very good lower bound on their minimum distances and contain distance-optimal codes. The other two families of binary cyclic codes are composed of binary duadic codes with a square-root-like lower bound on their minimum distances. As a by-product, two infinite families of self-dual binary codes with a square-root-like lower bound on their minimum distances are obtained.
Reinforcement learning for enhanced online gradient-based parameter adaptation in metaheuristics

Parameter setting is a significant and demanding process in metaheuristics. It requires deep understanding of the considered algorithm as well as of the considered optimization problem. Thus, it has attracted ongoing scientific interest counting a considerable number of relevant approaches, which are mostly focused in offline and ad hoc procedures. Interestingly, the parameter tuners contain also their own parameters. It is a basic requirement that these parameters shall be easily tunable and mildly affecting the tuner’s performance. The present work proposes an enhanced version of the recently proposed gradient-based parameter adaptation method, which offers an online general-purpose parameter tuning procedure that is applicable to any metaheuristic. The proposed method dynamically tunes the tuner’s parameters by using the REINFORCE reinforcement learning paradigm. The method is demonstrated on the differential evolution algorithm for established test suites of variable dimensions as well as for the real-world application of detecting atomic clusters through the minimization of Lennard-Jones potentials. The obtained results are statistically analyzed and compared with state-of-the-art auto-tuning algorithms. The analysis reveals the promising performance of the proposed approach as well as its robustness under the considered experimental setting.
Harnessing exciton-polaritons for digital computing, neuromorphic computing, and optimization [Invited]

Polaritons are quasiparticles resulting from the strong quantum coupling of light and matter. Peculiar properties of polaritons are a mixture of physics usually restricted to one of these realms, making them interesting for study not only from the fundamental point of view but also for applications. In recent years, many studies have been devoted to the potential use of exciton-polaritons for computing. Very recently, it has been shown experimentally that they can be harnessed not only for digital computing but also for optical neural networks and for optimization related to hard computational problems. Here, we provide a brief review of recent studies and the most important results in this area. We focus our attention, in particular, on the emerging concepts of non-von-Neumann computing schemes and their realizations in exciton-polariton systems.
Towards Disentangling Relevance and Bias in Unbiased Learning to Rank

Unbiased learning to rank (ULTR) studies the problem of mitigating various biases from implicit user feedback data such as clicks, and has been receiving considerable attention recently. A popular ULTR approach for real-world applications uses a two-tower architecture, where click modeling is factorized into a relevance tower with regular input features, and a bias tower with bias-relevant inputs such as the position of a document. A successful factorization will allow the relevance tower to be exempt from biases. In this work, we identify a critical issue that existing ULTR methods ignored - the bias tower can be confounded with the relevance tower via the underlying true relevance. In particular, the positions were determined by the logging policy, i.e., the previous production model, which would possess relevance information. We give both theoretical analysis and empirical results to show the negative effects on relevance tower due to such a correlation. We then propose two methods to mitigate the negative confounding effects by better disentangling relevance and bias. Offline empirical results on both controlled public datasets and a large-scale industry dataset show the effectiveness of the proposed approaches. We conduct a live experiment on a popular web store for four weeks, and find a significant improvement in user clicks over the baseline, which ignores the negative confounding effect.
A Lightweight, Efficient and Explainable-by-Design Convolutional Neural Network for Internet Traffic Classification

Traffic classification, i.e., the identification of the type of applications flowing in a network, is a strategic task for numerous activities (e.g., intrusion detection, routing). This task faces some critical challenges that current deep learning approaches do not address. The design of current approaches do not take into consideration the fact that networking hardware (e.g., routers) often runs with limited computational resources. Further, they do not meet the need for faithful explainability highlighted by regulatory bodies. Finally, these traffic classifiers are evaluated on small datasets which fail to reflect the diversity of applications in real-world settings.
Attribute subspaces for zero-shot learning

Zero-shot learning (ZSL) aims to recognize unseen categories without corresponding training samples, which is a practical yet challenging task in computer vision and pattern recognition community. Current state-of-the-art locality-based ZSL methods aim to learn the explicit locality of discriminative attributes, which may suffer from insufficient class-level attribute supervision. In this paper, we introduce an Attribute Subspace learning method for ZSL (AS-ZSL) to learn implicit attribute composition, which is more general than attribute localization with only class-level attribute supervision. AS-ZSL exploits subspace representations that can effectively capture the intrinsic composition of high-dimensional image features and the diversity within attribute appearance. Furthermore, we develop a subspace distance based triplet loss to improve the distinguishability of the attribute subspace representation. Attribute subspace learning module is only needed for the training phase to jointly learn discriminative global features. This leads to a compact inference phase. Furthermore, the proposed AS-ZSL can be naturally extended to adapt to the transductive ZSL setting using a novel self-supervised training strategy. Extensive experimental results on several widely used ZSL datasets, i.e., CUB, AwA2, and SUN, demonstrate the advantage of AS-ZSL compared with the state-of-the-art under different ZSL settings.
Less is More: Summary of Long Instructions is Better for Program Synthesis

Despite the success of large pre-trained language models (LMs) such as Codex, they show below-par performance on the larger and more complicated programming related questions.We show that LMs benefit from the summarized version of complicated questions.Our findings show that superfluous information often present in problem description such as human characters, background stories, and names (which are included to help humans in understanding a task) does not help models in understanding a task.To this extent, we create a metadataset from the frequently used APPS dataset and the newly created CodeContests dataset for the program synthesis task.Our meta-dataset consists of human and synthesized summaries of the long and complicated programming questions.Experimental results on Codex show that our proposed approach outperforms baseline by 8.13% on the APPS dataset and 11.88% on the CodeContests dataset on average in terms of strict accuracy.Our analysis shows that summaries significantly improve performance for introductory (9.86%) and interview (11.48%) programming questions.However, it shows improvement by a small margin (∼ 2%) for competitive programming questions, implying scope for future research in this direction.
FedDefender: Client-Side Attack-Tolerant Federated Learning

Federated learning enables learning from decentralized data sources without compromising privacy, which makes it a crucial technique. However, it is vulnerable to model poisoning attacks, where malicious clients interfere with the training process. Previous defense mechanisms have focused on the server-side by using careful model aggregation, but this may not be effective when the data is not identically distributed or when attackers can access the information of benign clients. In this paper, we propose a new defense mechanism that focuses on the client-side, called FedDefender, to help benign clients train robust local models and avoid the adverse impact of malicious model updates from attackers, even when a server-side defense cannot identify or remove adversaries. Our method consists of two main components: (1) attack-tolerant local meta update and (2) attack-tolerant global knowledge distillation. These components are used to find noise-resilient model parameters while accurately extracting knowledge from a potentially corrupted global model. Our client-side defense strategy has a flexible structure and can work in conjunction with any existing server-side strategies. Evaluations of real-world scenarios across multiple datasets show that the proposed method enhances the robustness of federated learning against model poisoning attacks.
AdaProp: Learning Adaptive Propagation for Graph Neural Network based Knowledge Graph Reasoning

Due to the popularity of Graph Neural Networks (GNNs), various GNN-based methods have been designed to reason on knowledge graphs (KGs). An important design component of GNN-based KG reasoning methods is called the propagation path, which contains a set of involved entities in each propagation step. Existing methods use hand-designed propagation paths, ignoring the correlation between the entities and the query relation. In addition, the number of involved entities will explosively grow at larger propagation steps. In this work, we are motivated to learn an adaptive propagation path in order to filter out irrelevant entities while preserving promising targets. First, we design an incremental sampling mechanism where the nearby targets and layer-wise connections can be preserved with linear complexity. Second, we design a learning-based sampling distribution to identify the semantically related entities. Extensive experiments show that our method is powerful, efficient and semantic-aware. The code is available at https://github.com/LARS-research/AdaProp.