Reviewerly/exHarmony huggingface dataset:
SonicASL

We propose SonicASL, a real-time gesture recognition system that can recognize sign language gestures on the fly, leveraging front-facing microphones and speakers added to commodity earphones worn by someone facing the person making the gestures. In a user study (N=8), we evaluate the recognition performance of various sign language gestures at both the word and sentence levels. Given 42 frequently used individual words and 30 meaningful sentences, SonicASL can achieve an accuracy of 93.8% and 90.6% for word-level and sentence-level recognition, respectively. The proposed system is tested in two real-world scenarios: indoor (apartment, office, and corridor) and outdoor (sidewalk) environments with pedestrians walking nearby. The results show that our system can provide users with an effective gesture recognition tool with high reliability against environmental factors such as ambient noises and nearby pedestrians.
Consistent Instance False Positive Improves Fairness in Face Recognition

Demographic bias is a significant challenge in practical face recognition systems. Existing methods heavily rely on accurate demographic annotations. However, such annotations are usually unavailable in real scenarios. Moreover, these methods are typically designed for a specific demographic group and are not general enough. In this paper, we propose a false positive rate penalty loss, which mitigates face recognition bias by increasing the consistency of instance False Positive Rate (FPR). Specifically, we first define the instance FPR as the ratio between the number of the non-target similarities above a unified threshold and the total number of the non-target similarities. The unified threshold is estimated for a given total FPR. Then, an additional penalty term, which is in proportion to the ratio of instance FPR overall FPR, is introduced into the denominator of the softmax-based loss. The larger the instance FPR, the larger the penalty. By such unequal penalties, the instance FPRs are supposed to be consistent. Compared with the previous debiasing methods, our method requires no demographic annotations. Thus, it can mitigate the bias among demographic groups divided by various attributes, and these attributes are not needed to be previously predefined during training. Extensive experimental results on popular benchmarks demonstrate the superiority of our method over state-of-the-art competitors. Code and pre-trained models are available at https://github.com/xkx0430/FairnessFR.
Automated Assessment of Question Quality on Online Community Forums

People around the world often rely on online community forums for answers to their queries. These forums have become hugely popular in the last decade, leading to a spurt in the number of users and questions. For a better user experience, quality monitoring is essential. However, manual moderation of millions of questions is infeasible. Prior works mostly rely on handcrafted features which is ineffective or use community feedback as part of learning which makes them unsuitable for monitoring during question creation. In this work, we use recent deep learning techniques to assess the quality of questions in online community forums at creation time. We evaluate our model on the StackOverflow dataset that contains 60000 questions across three qualities. Our model achieves an F1 score of 0.92 on this dataset.
Extreme Learning Machine Applied to Software Development Effort Estimation

The project management process has been used in the
Coarse-Fine Networks for Temporal Activity Detection in Videos

In this paper, we introduce Coarse-Fine Networks, a twostream architecture which benefits from different abstractions of temporal resolution to learn better video representations for long-term motion. Traditional Video models process inputs at one (or few) fixed temporal resolution without any dynamic frame selection. However, we argue that, processing multiple temporal resolutions of the input and doing so dynamically by learning to estimate the importance of each frame can largely improve video representations, specially in the domain of temporal activity localization. To this end, we propose (1) ‘Grid Pool’, a learned temporal downsampling layer to extract coarse features, and, (2) ‘Multi-stage Fusion’, a spatio-temporal attention mechanism to fuse a finegrained context with the coarse features. We show that our method outperforms the state-of-the-arts for action detection in public datasets including Charades with a significantly reduced compute and memory footprint. The code is available at https://github.com/kkahatapitiya/Coarse-Fine-Networks.
Removing Diffraction Image Artifacts in Under-Display Camera via Dynamic Skip Connection Network

Development of Under-Display Camera (UDC) systems provides a true bezel-less and notch-free viewing experience on smartphones (and TV, laptops, tablets), while allowing images to be captured from the selfie camera embedded underneath. In a typical UDC system, the microstructure of the semi-transparent organic light-emitting diode (OLED) pixel array attenuates and diffracts the incident light on the camera, resulting in significant image quality degradation. Oftentimes, noise, flare, haze, and blur can be observed in UDC images. In this work, we aim to analyze and tackle the aforementioned degradation problems. We define a physics-based image formation model to better understand the degradation. In addition, we utilize one of the world’s first commodity UDC smartphone prototypes to measure the real-world Point Spread Function (PSF) of the UDC system, and provide a model-based data synthesis pipeline to generate realistically degraded images. We specially design a new domain knowledge-enabled Dynamic Skip Connection Network (DISCNet) to restore the UDC images. We demonstrate the effectiveness of our method through extensive experiments on both synthetic and real UDC data. Our physics-based image formation model and proposed DISCNet can provide foundations for further exploration in UDC image restoration, and even for general diffraction artifact removal in a broader sense. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
Learning Camera Localization via Dense Scene Matching

Camera localization aims to estimate 6 DoF camera poses from RGB images. Traditional methods detect and match interest points between a query image and a prebuilt 3D model. Recent learning-based approaches encode scene structures into a specific convolutional neural network (CNN) and thus are able to predict dense coordinates from RGB images. However, most of them require re-training or re-adaption for a new scene and have difficulties in handling large-scale scenes due to limited network capacity. We present a new method for scene agnostic camera localization using dense scene matching (DSM), where a cost volume is constructed between a query image and a scene. The cost volume and the corresponding coordinates are processed by a CNN to predict dense coordinates. Camera poses can then be solved by PnP algorithms. In addition, our method can be extended to temporal domain, which leads to extra performance boost during testing time. Our scene-agnostic approach achieves comparable accuracy as the existing scene-specific approaches, such as KFNet, on the 7scenes and Cambridge benchmark. This approach also remarkably outperforms state-of-the-art scene-agnostic dense coordinate regression network SANet. The Code is available at https://github.com/Tangshitao/DenseScene-Matching.
A High-Precision Fast Smoky Vehicle Detection Method Based on Improved Yolov5 Network

A high-precision and fast smoky vehicle detection method was proposed. Since the existing target detection models deployed on embedded devices cannot meet the needs of rapid detection, an improved lightweight network based on Yolov5 was adopted in this paper. The backbone of Yolov5s was improved by Mobilenetv3-small to reduce the number of model parameters and calculations. In order to detect motor vehicle exhaust with high precision, a vehicle exhaust dataset is collected and established. Due to the interference of vehicle shadows and the occlusion between vehicles, Cutout and saturation transformation were applied to expand the self-built dataset, which was finally expanded to 6102 images. Experiments results show that after using data augmentation, the detection accuracy is increased by 8.5%. The improved network is deployed on embedded devices, and the detection speed of the network can reach 12.5FPS, which is 2 times higher than Yolov5's. The amount of improved network parameters is only 0.48M. This research proposes an efficient target detection model, and provides a possible method for the development of low-cost and rapid vehicle exhaust detection equipment.
Occlusion Guided Scene Flow Estimation on 3D Point Clouds

3D scene flow estimation is a vital tool in perceiving our environment given depth or range sensors. Unlike optical flow, the data is usually sparse and in most cases partially occluded in between two temporal samplings. Here we propose a new scene flow architecture called OGSF-Net which tightly couples the learning for both flow and occlusions between frames. Their coupled symbiosis results in a more accurate prediction of flow in space. Unlike a traditional multi-action network, our unified approach is fused throughout the network, boosting performances for both occlusion detection and flow estimation. Our architecture is the first to gauge the occlusion in 3D scene flow estimation on point clouds. In key datasets such as Flyingthings3D and KITTI, we achieve the state-of-the-art results. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1 2</sup>
An Efficient and Secure Multimessage and Multireceiver Signcryption Scheme for Edge-Enabled Internet of Vehicles

The Internet of Vehicles (IoV) is considered an enhancement of existing vehicular ad-hoc networks, which helps connect mobile vehicles to the Internet of Things (IoT) with the support of 5G networks. To assure the quality-of-service demand by the users, the edge computing paradigm of 5G networks can be incorporated in the IoV environment for supporting compute-intensive applications. The basic safety messages are typically transmitted using a multicast pattern in the IoV-enabled edge computing paradigm. The use of the multicast channel may accelerate the communication process; however, it is prone to various attacks due to the open nature of wireless networks. This article proposes a multimessage and multireceiver signcryption scheme for the multicast channel in a certificateless setting to solve the key escrow problem. The security of the partial private key is dependent on the secure channel, which increases the complexities of the system. Therefore, in the proposed scheme, we introduce a new idea that does not require a secure channel. The key generation center only sends the pseudo partial private key of the users on a public channel. Furthermore, the proposed scheme is based on hyper-elliptic curve cryptography (HECC), which has much smaller key sizes as compared to elliptic curve cryptography (ECC). The security proofs and performance comparison for our scheme are carried out. The findings show that the proposed scheme provides high security while using less computational and communication costs.
Ensembling with Deep Generative Views

Recent generative models can synthesize "views" of artificial images that mimic real-world variations, such as changes in color or pose, simply by learning from unlabeled image collections. Here, we investigate whether such views can be applied to real images to benefit downstream analysis tasks such as image classification. Using a pre-trained generator, we first find the latent code corresponding to a given real input image. Applying perturbations to the code creates natural variations of the image, which can then be ensembled together at test-time. We use StyleGAN2 as the source of generative augmentations and investigate this setup on classification tasks involving facial attributes, cat faces, and cars. Critically, we find that several design decisions are required towards making this process work; the perturbation procedure, weighting between the augmentations and original image, and training the classifier on synthesized images can all impact the result. Currently, we find that while test-time ensembling with GAN-based augmentations can offer some small improvements, the remaining bottlenecks are the efficiency and accuracy of the GAN reconstructions, coupled with classifier sensitivities to artifacts in GAN-generated images.
Zero-Shot Instance Segmentation

Deep learning has significantly improved the precision of instance segmentation with abundant labeled data. However, in many areas like medical and manufacturing, collecting sufficient data is extremely hard and labeling this data requires high professional skills. We follow this motivation and propose a new task set named zero-shot instance segmentation (ZSI). In the training phase of ZSI, the model is trained with seen data, while in the testing phase, it is used to segment all seen and unseen instances. We first formulate the ZSI task and propose a method to tackle the challenge, which consists of Zero-shot Detector, Semantic Mask Head, Background Aware RPN and Synchronized Background Strategy. We present a new benchmark for zero-shot instance segmentation based on the MS-COCO dataset. The extensive empirical results in this benchmark show that our method not only surpasses the state-of-the-art results in zero-shot object detection task but also achieves promising performance on ZSI. Our approach will serve as a solid baseline and facilitate future research in zero-shot instance segmentation. Code available at $\color{Magenta}{\text{ZSI}}$.
Sentiment Analysis Using XLM-R Transformer and Zero-shot Transfer Learning on Resource-poor Indian Language

Sentiment analysis on social media relies on comprehending the natural language and using a robust machine learning technique that learns multiple layers of representations or features of the data and produces state-of-the-art prediction results. The cultural miscellanies, geographically limited trending topic hash-tags, access to aboriginal language keyboards, and conversational comfort in native language compound the linguistic challenges of sentiment analysis. This research evaluates the performance of cross-lingual contextual word embeddings and zero-shot transfer learning in projecting predictions from resource-rich English to resource-poor Hindi language. The cross-lingual XLM-RoBERTa classification model is trained and fine-tuned using the English language Benchmark SemEval 2017 dataset Task 4 A and subsequently zero-shot transfer learning is used to evaluate the classification model on two Hindi sentence-level sentiment analysis datasets, namely, IITP-Movie and IITP-Product review datasets. The proposed model compares favorably to state-of-the-art approaches and gives an effective solution to sentence-level (tweet-level) analysis of sentiments in a resource-poor scenario. The proposed model compares favorably to state-of-the-art approaches and achieves an average performance accuracy of 60.93 on both the Hindi datasets.
Privacy and Artificial Intelligence

Artificial intelligence is a rapidly developing field of research with many practical applications. Congruent to advances in technologies that enable big data, deep learning, and neural networks to train, learn, and predict, artificial intelligence creates new risks that are difficult to predict and manage. Such risks include economic turmoil, existential crises, and the dissolution of individual privacy. If unchecked, the capabilities of artificially intelligent systems could pose a fundamental threat to privacy in their operation or these systems may leak information under adversarial conditions. In this article, we survey the literature and provide various scenarios for the use of artificial intelligence, highlighting potential risks to privacy and offering various mitigating strategies. For the purpose of this research, a North American perspective of privacy is adopted. Impact statement-While an appreciation of the privacy risks associated with artificial intelligence is important, a thorough understanding of the assortment of different technologies that comprise artificial intelligence better prepares those implementing such systems in assessing privacy impacts. This can be achieved through the independent consideration of each constituent of an artificially intelligent system and its interactions. Under individual consideration, privacy-enhancing tools can be applied in a targeted manner to reduce the risk associated with specific components of an artificially intelligent system. A generalized North American approach to assess privacy risks in such systems is proposed that will retain applicability as the field of research evolves and can be adapted to account for various sociopolitical influences. With such an approach, privacy risks in artificial intelligent systems can be well understood, measured, and reduced.
Probing the buckling of pressurized spherical shells

The prediction of the critical buckling conditions of shell structures is plagued by imperfection sensitivity. Non-destructive testing through point-load probing has been recently proposed to map the stability landscape of cylindrical shells. However, the counterpart procedure for spherical shells is still debatable. Here, we focus on the mechanical response of pressurized spherical shells containing a single dimple–like defect to a point probe. Combining experiments, finite element modeling, and existing results from classic shell theory, we characterize the nonlinear force-indentation response of imperfect shells at different pressurization levels. From these curves, we seek to identify the critical buckling pressure of the shell. In particular, the indentation angle is varied systematically to examine its effect on the probing efficacy. We find that the critical buckling point can be inferred non-destructively by tracking the maxima of the indentation force–displacement curves, if the probe is implemented sufficiently close to the defect. When probing further away from the defect, the test fails in predicting the onset of buckling since the deformation due to indentation remains localized in the vicinity of the probe. Using FEM simulations and shallow shell theory, we quantify the characteristic length associated with this localized deformation, both in the linear and nonlinear regimes. Our results demonstrate the limiting conditions of applicability for the usage of probing as a non-destructive technique to assess the stability of spherical shells.
Weakly Convex Optimization over Stiefel Manifold Using Riemannian Subgradient-Type Methods

We consider a class of nonsmooth optimization problems over the Stiefel manifold, in which the objective function is weakly convex in the ambient Euclidean space. Such problems are ubiquitous in engineering applications but still largely unexplored. We present a family of Riemannian subgradient-type methods---namely Riemannian subgradient, incremental subgradient, and stochastic subgradient methods---to solve these problems and show that they all have an iteration complexity of $\mathcal{O}(\varepsilon^{-4})$ for driving a natural stationarity measure below $\varepsilon$. In addition, we establish the local linear convergence of the Riemannian subgradient and incremental subgradient methods when the problem at hand further satisfies a sharpness property and the algorithms are properly initialized and use geometrically diminishing stepsizes. To the best of our knowledge, these are the first convergence guarantees for using Riemannian subgradient-type methods to optimize a class of nonconvex nonsmooth functions over the Stiefel manifold. The fundamental ingredient in the proof of the aforementioned convergence results is a new Riemannian subgradient inequality for restrictions of weakly convex functions on the Stiefel manifold, which could be of independent interest. We also show that our convergence results can be extended to handle a class of compact embedded submanifolds of the Euclidean space. Finally, we discuss the sharpness properties of various formulations of the robust subspace recovery and orthogonal dictionary learning problems and demonstrate the convergence performance of the algorithms on both problems via numerical simulations.
Generalization on Unseen Domains via Inference-time Label-Preserving Target Projections

Generalization of machine learning models trained on a set of source domains on unseen target domains with different statistics, is a challenging problem. While many approaches have been proposed to solve this problem, they only utilize source data during training but do not take advantage of the fact that a single target example is available at the time of inference. Motivated by this, we propose a method that effectively uses the target sample during inference beyond mere classification. Our method has three components - (i) A label-preserving feature or metric transformation on source data such that the source samples are clustered in accordance with their class irrespective of their domain (ii) A generative model trained on the these features (iii) A label-preserving projection of the target point on the source-feature manifold during inference via solving an optimization problem on the input space of the generative model using the learned metric. Finally, the projected target is used in the classifier. Since the projected target feature comes from the source manifold and has the same label as the real target by design, the classifier is expected to perform better on it than the true target. We demonstrate that our method outperforms the state-of-the-art Domain Generalization methods on multiple datasets and tasks.
MultiBodySync: Multi-Body Segmentation and Motion Estimation via 3D Scan Synchronization

We present MultiBodySync, a novel, end-to-end trainable multi-body motion segmentation and rigid registration framework for multiple input 3D point clouds. The two non-trivial challenges posed by this multi-scan multibody setting that we investigate are: (i) guaranteeing correspondence and segmentation consistency across multiple input point clouds capturing different spatial arrangements of bodies or body parts; and (ii) obtaining robust motion-based rigid body segmentation applicable to novel object categories. We propose an approach to address these issues that incorporates spectral synchronization into an iterative deep declarative network, so as to simultaneously recover consistent correspondences as well as motion segmentation. At the same time, by explicitly disentangling the correspondence and motion segmentation estimation modules, we achieve strong generalizability across different object categories. Our extensive evaluations demonstrate that our method is effective on various datasets ranging from rigid parts in articulated objects to individually moving objects in a 3D scene, be it single-view or full point clouds. Code at https://github.com/huangjh-pub/multibody-sync.