Reviewerly/exHarmony huggingface dataset:
R-YOLO: A Robust Object Detector in Adverse Weather

Learning a robust object detector in adverse weather with real-time efficiency is of great importance for the visual perception task for autonomous driving systems. In this article, we propose a framework to improve the YOLO to a robust detector, denoted as R(obust)-YOLO, without the need for annotations in adverse weather. Considering the distribution gap between the normal weather images and the adverse weather images, our framework consists of an image quasi-translation network (QTNet) and a feature calibration network (FCNet) for adapting the normal weather domain to the adverse weather domain gradually. Specifically, we use the simple yet effective QTNet for generating images that inherit the annotations in the normal weather domain and interpolate the gap between the two domains. Then, in FCNet, we propose two kinds of adversarial-learning-based feature calibration modules to effectively align the feature representations in two domains in a local-to-global manner. With such a learning framework, our R-YOLO does not change the original YOLO structure, and thus it is applicable to all the YOLO-series detectors. Extensive experimental results of our R-YOLOv3, R-YOLOv5, and R-YOLOX on both the hazy and rainy datasets show that our method outperforms other detectors with dehaze/derain as the preprocessing step and other unsupervised domain adaptation (UDA)-based detectors, which confirms the effectiveness of our method on improving the robustness by only leveraging the unlabeled adverse weather images. Our code and pretrained models are available at: <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/qinhongda8/R-YOLO</uri> .
A Large-scale Comprehensive Dataset and Copy-overlap Aware Evaluation Protocol for Segment-level Video Copy Detection

In this paper, we introduce VCSL (Video Copy Segment Localization), a new comprehensive segment-level annotated video copy dataset. Compared with existing copy detection datasets restricted by either video-level annotation or small-scale, VCSL not only has two orders of magnitude more segment-level labelled data, with 160k realistic video copy pairs containing more than 280k localized copied segment pairs, but also covers a variety of video categories and a wide range of video duration. All the copied segments inside each collected video pair are manually extracted and accompanied by precisely annotated starting and ending timestamps. Alongside the dataset, we also propose a novel evaluation protocol that better measures the prediction accuracy of copy overlapping segments between a video pair and shows improved adaptability in different scenarios. By benchmarking several baseline and state-of-the-art segment-level video copy detection methods with the proposed dataset and evaluation metric, we provide a comprehensive analysis that uncovers the strengths and weaknesses of current approaches, hoping to open up promising directions for future works. The VCSL dataset, metric and benchmark codes are all publicly available at https://github.com/alipay/vCSL.
Applied trajectory generation to dock a feeder vessel

Autonomous sailing is seen as one of the possible solutions to cope with the decrease in qualified officers, to minimise the risk to humans and ships at challenging conditions, and to decrease the environmental impact of the transport sector. Autonomous sailing is not limited to moving the vessel safely through the seas, but it also includes docking the vessel. Docking is considered a difficult and critical manoeuvre in autonomous sailing. The objective of this work is to construct a trajectory including its time derivatives for our newly designed 71m long underactuated feeder vessel to approach a dock. The trajectory can be used by a feedback controller to steer the vessel to the dock, and the time derivatives can be used to construct a feed forward signal to improve the tracking performance. An optimal control problem is formulated to answer i) what the best pose is to start an approach to the dock, and ii) what trajectory is optimal from an arbitrary initial pose. The resulting trajectory gives an optimal minimal-time solution, but the computational time was too large to employ real-time. A second trajectory was constructed based on Bézier curves. The computational load of this trajectory is negligible. The trajectory can be used for a subset of initial states. It does provide a smooth trajectory with all the derivatives, and is seen as a good option when the vessel starts the docking manoeuvre from a not too challenging pose.
Long-Tailed Instance Segmentation Using Gumbel Optimized Loss

Major advancements have been made in the field of object detection and segmentation recently. However, when it comes to rare categories, the state-of-the-art methods fail to detect them, resulting in a significant performance gap between rare and frequent categories. In this paper, we identify that Sigmoid or Softmax functions used in deep detectors are a major reason for low performance and are sub-optimal for long-tailed detection and segmentation. To address this, we develop a Gumbel Optimized Loss (GOL), for long-tailed detection and segmentation. It aligns with the Gumbel distribution of rare classes in imbalanced datasets, considering the fact that most classes in long-tailed detection have low expected probability. The proposed GOL significantly outperforms the best state-of-the-art method by $$1.1\%$$ on AP, and boosts the overall segmentation by $$9.0\%$$ and detection by $$8.0\%$$ , particularly improving detection of rare classes by $$20.3\%$$ , compared to Mask-RCNN, on LVIS dataset. Code available at: https://github.com/kostas1515/GOL .
Dynamic NLP Enabled Chatbot for Rural Health Care in India

Medical practitioners can use telemedicine to connect with their patients during the recent Coronavirus outbreak, while also attempting to prevent COVID-19 transmission among patients and clinicians. In the middle of the pandemic, Telemedicine can help patients by allowing them to obtain supportive care without having to visit a hospital via a conversational AI-based application. As a result, telehealth will transform in-person care into remote patient connection fast and drastically. In this paper, Natural Language Processing (NLP)-based Multilingual Conversational Bot has been developed to provide chronic patients with free fundamental healthcare education, information, and guidance. Countries like India, rural healthcare management is challenging task with available medical experts and equipment’s. So proposed Tele-health care management leads towards healthy rural people and their development. Existing Application Programming Interface (API) like Dialogflow can be used along with our healthcare mobile application to provide live treatment and diagnosis.
Proto2Proto: Can you recognize the car, the way I do?

Prototypical methods have recently gained a lot of attention due to their intrinsic interpretable nature, which is obtained through the prototypes. With growing use cases of model reuse and distillation, there is a need to also study transfer of interpretability from one model to another. We present Proto2Proto, a novel method to transfer interpretability of one prototypical part network to another via knowledge distillation. Our approach aims to add interpretability to the “dark” knowledge transferred from the teacher to the shallower student model. We propose two novel losses: “Global Explanation” loss and “Patch-Prototype Correspondence” loss to facilitate such a transfer. Global Explanation loss forces the student prototypes to be close to teacher prototypes, and Patch-Prototype Cor-respondence loss enforces the local representations of the student to be similar to that of the teacher. Further, we propose three novel metrics to evaluate the student's proximity to the teacher as measures of interpretability transfer in our settings. We qualitatively and quantitatively demon-strate the effectiveness of our method on CUB-200-2011 and Stanford Cars datasets. Our experiments show that the proposed method indeed achieves interpretability transfer from teacher to student while simultaneously exhibiting competitive performance. The code is available at h t tps: //github.com/archmaester/proto2proto
Does text attract attention on e-commerce images: A novel saliency prediction dataset and method

E-commerce images are playing a central role in attracting people's attention when retailing and shopping online, and an accurate attention prediction is of significant importance for both customers and retailers, where its research is yet to start. In this paper, we establish the first dataset of saliency e-commerce images (SalECI), which allows for learning to predict saliency on the e-commerce images. We then provide specialized and thorough analysis by high-lighting the distinct features of e-commerce images, e.g., non-locality and correlation to text regions. Correspondingly, taking advantages of the non-local and self-attention mechanisms, we propose a salient SWin-Transformer back-bone, followed by a multi-task learning with saliency and text detection heads, where an information flow mechanism is proposed to further benefit both tasks. Experimental results have verified the state-of-the-art performances of our work in the e-commerce scenario.
Contrastive Haze-Aware Learning for Dynamic Remote Sensing Image Dehazing

Image dehazing methods aim to recover a clear image from its hazy counterpart. While various dehazing methods have been proposed, their performance on real-world remote sensing (RS) images remains unsatisfying. A key reason is that the complex weather and imaging conditions (e.g., large fields of view) cause the haze condition to dramatically change in different images, while most existing methods fail to flexibly adapt their dehazing model to the specific haze condition in each image. To mitigate this problem, we present a contrastive haze-aware learning based dynamic dehazing method which demonstrates two aspects of advantage. On one hand, a contrastive clustering scheme is utilized to learn the image-wise haze representation using a set of real-world hazy images in an unsupervised manner, which enables identifying and discriminating the specific haze condition in each given hazy image. On the other hand, with the learned haze representation, a parameter generator can produce haze-aware parameters to dynamically construct a dehazing model for the given hazy image, which empowers us to adaptively dehaze the image based on its specific haze condition and thus improves the generalization ability. In addition, a new contrastive loss defined based on the learned haze representation is further utilized for model training and leads to better performance. To demonstrate the effectiveness of the proposed method, we evaluate it on two benchmark RS image datasets including various real-world hazy images, and observe obviously superiority over other state-of-the-art competitors.
Iwin: Human-Object Interaction Detection via Transformer with Irregular Windows

This paper presents a new vision Transformer, named Iwin Transformer, which is specifically designed for human-object interaction (HOI) detection, a detailed scene understanding task involving a sequential process of human/object detection and interaction recognition. Iwin Transformer is a hierarchical Transformer which progressively performs token representation learning and token agglomeration within irregular windows. The irregular windows, achieved by augmenting regular grid locations with learned offsets, 1) eliminate redundancy in token representation learning, which leads to efficient human/object detection, and 2) enable the agglomerated tokens to align with humans/objects with different shapes, which facilitates the acquisition of highly-abstracted visual semantics for interaction recognition. The effectiveness and efficiency of Iwin Transformer are verified on the two standard HOI detection benchmark datasets, HICO-DET and V-COCO. Results show our method outperforms existing Transformers-based methods by large margins (3.7 mAP gain on HICO-DET and 2.0 mAP gain on V-COCO) with fewer training epochs ( $$0.5 \times $$ ).
Towards Lifelong Federated Learning in Autonomous Mobile Robots with Continuous Sim-to-Real Transfer

The role of deep learning (DL) in robotics has significantly deepened over the last decade. Intelligent robotic systems today are highly connected systems that rely on DL for a variety of perception, control and other tasks. At the same time, autonomous robots are being increasingly deployed as part of fleets, with collaboration among robots becoming a more relevant factor. From the perspective of collaborative learning, federated learning (FL) enables continuous training of models in a distributed, privacy-preserving way. This paper focuses on vision-based obstacle avoidance for mobile robot navigation. On this basis, we explore the potential of FL for distributed systems of mobile robots enabling continuous learning via the engagement of robots in both simulated and real-world scenarios. We extend previous works by studying the performance of different image classifiers for FL, compared to centralized, cloud-based learning with a priori aggregated data. We also introduce an approach to continuous learning from mobile robots with extended sensor suites able to provide automatically labelled data while they are completing other tasks. We show that higher accuracies can be achieved by training the models in both simulation and reality, enabling continuous updates to deployed models.
Zero-Query Transfer Attacks on Context-Aware Object Detectors

Adversarial attacks perturb images such that a deep neural network produces incorrect classification results. A promising approach to defend against adversarial attacks on natural multi-object scenes is to impose a context-consistency check, wherein, if the detected objects are not consistent with an appropriately defined context, then an attack is suspected. Stronger attacks are needed to fool such context-aware detectors. We present the first approach for generating context-consistent adversarial attacks that can evade the context-consistency check of black-box object detectors operating on complex, natural scenes. Unlike many black-box attacks that perform repeated attempts and open themselves to detection, we assume a “zero-query” setting, where the attacker has no knowledge of the classification decisions of the victim system. First, we derive multiple attack plans that assign incorrect labels to victim objects in a context-consistent manner. Then we design and use a novel data structure that we call the perturbation success probability matrix, which enables us to filter the attack plans and choose the one most likely to succeed. This final attack plan is implemented using a perturbation-bounded adversarial attack algorithm. We compare our zero-query attack against a few-query scheme that repeatedly checks if the victim system is fooled. We also compare against state-of-the-art context-agnostic attacks. Against a context-aware defense, the fooling rate of our zero-query approach is significantly higher than context-agnostic approaches and higher than that achievable with up to three rounds of the fewquery scheme.
Explaining Deep Convolutional Neural Networks via Latent Visual-Semantic Filter Attention

Interpretability is an important property for visual mod-els as it helps researchers and users understand the in-ternal mechanism of a complex model. However, gener-ating semantic explanations about the learned representation is challenging without direct supervision to produce such explanations. We propose a general framework, La-tent Visual Semantic Explainer (LaViSE), to teach any ex-isting convolutional neural network to generate text de-scriptions about its own latent representations at the filter level. Our method constructs a mapping between the vi-sual and semantic spaces using generic image datasets, using images and category names. It then transfers the map-ping to the target domain which does not have semantic la-bels. The proposedframework employs a modular structure and enables to analyze any trained network whether or not its original training data is available. We show that our method can generate novel descriptions for learned filters beyond the set of categories defined in the training dataset and perform an extensive evaluation on multiple datasets. We also demonstrate a novel application of our method for unsupervised dataset bias analysis which allows us to auto-matically discover hidden biases in datasets or compare dif-ferent subsets without using additional labels. The dataset and code are made public to facilitate further research. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> https://github.com/YuYang0901/LaViSE
UPLIFT

Data science pipelines are typically exploratory. An integral task of such pipelines are feature transformations, which transform raw data into numerical matrices or tensors for training or scoring. There exist a wide variety of transformations for different data modalities. These feature transformations incur large computational overhead due to expensive string processing and dictionary creation. Existing ML systems address this overhead by static parallelization schemes and interleaving transformations with model training. These approaches show good performance improvements for simple transformations, but struggle to handle different data characteristics (many features/distinct items) and multi-pass transformations. A key observation is that good parallelization strategies for feature transformations depend on data characteristics. In this paper, we introduce UPLIFT, a framework for P aralle LI zing F eature T ransformations. UPLIFT constructs a fine-grained task graph for a set of transformations, optimizes the plan according to data characteristics, and executes this plan in a cache-conscious manner. We show that the resulting framework is applicable to a wide range of transformations. Furthermore, we propose the FTBench benchmark with transformations and datasets from various domains. On this benchmark, UPLIFT yields speedups of up to 31.6x (9.27x on average) compared to state-of-the-art ML systems.
Explainable Analysis of Deep Learning Methods for Sar Image Classification

Deep learning methods exhibit outstanding performance in synthetic aperture radar (SAR) image interpretation tasks. However, these are black box models that limit the com-prehension of their predictions. Therefore, to meet this challenge, we have utilized explainable artificial intelli-gence (XAI) methods for the SAR image classification task. Specifically, we trained state-of-the-art convolutional neural networks for each polarization format on OpenSARUrban dataset and then investigate eight explanation methods to analyze the predictions of the CNN classifiers of SAR images. These XAI methods are also evaluated qualitatively and quantitatively which shows that Occlusion achieves the most reliable interpretation performance in terms of Max-Sensitivity but with a low-resolution explanation heatmap. The explanation results provide some insights into the in-ternal mechanism of black-box decisions for SAR image classification.
A Reconfigurable Depth-Wise Convolution Module for Heterogeneously Quantized DNNs

In Deep Neural Networks (DNN), the depth-wise separable convolution has often replaced the standard 2D convolution having much fewer parameters and operations. Another common technique to squeeze DNNs is heterogeneous quantization, which uses a different bitwidth for each layer. In this context we propose for the first time a novel Reconfigurable Depth-wise convolution Module (RDM), which uses multipliers that can be reconfigured to support 1, 2 or 4 operations at the same time at increasingly lower precision of the operands. We leveraged High Level Synthesis to produce five RDM variants with different channels parallelism to cover a wide range of DNNs. The comparisons with a non-configurable Standard Depth-wise convolution module (SDM) on a CMOS FDSOI 28-nm technology show a significant latency reduction for a given silicon area for the low-precision configurations.
An introduction to radical participatory design: decolonising participatory design processes

Abstract Outside of community-led design projects, most participatory design processes initiated by a company or organisation maintain or even strengthen power imbalances between the design organisation and the community on whose purported behalf they are designing, further increasing the absencing experience. Radical participatory design (RPD) is a radically relational answer to the coloniality inherent in participatory design where the community members’ disappointment is greater due to the greater expectations and presencing potential of a ‘participatory design’ process. We introduce the term RPD to show how research and design processes can be truly participatory to the root or core. Instead of treating participatory design as a method, a way of conducting a method, or a methodology, we introduce RPD as a meta-methodology, a way of doing any methodology. We explicitly describe what participation means and compare and contrast design processes based on the amount of participation, creating a typology of participation. We introduce ‘designer as community member’, ‘community member as designer,’ and ‘community member as facilitator’ models and provide characteristics for the meta-methodology of RPD.
Modeling sRGB Camera Noise with Normalizing Flows

Noise modeling and reduction are fundamental tasks in low-level computer vision. They are particularly important for smartphone cameras relying on small sensors that exhibit visually noticeable noise. There has recently been renewed interest in using data-driven approaches to improve camera noise models via neural networks. These data-driven approaches target noise present in the raw-sensor image before it has been processed by the camera's image signal processor (ISP). Modeling noise in the RAW-rgb domain is useful for improving and testing the in-camera denoising algorithm; however, there are situations where the camera's ISP does not apply denoising or additional denoising is desired when the RAW-rgb domain image is no longer available. In such cases, the sensor noise propagates through the ISP to the final rendered image encoded in standard RGB (sRGB). The nonlinear steps on the ISP culminate in a significantly more complex noise distribution in the sRGB domain and existing raw-domain noise models are unable to capture the sRGB noise distribution. We propose a new sRGB-domain noise model based on normalizing flows that is capable of learning the complex noise distribution found in sRGB images under various ISO levels. Our normalizing flows-based approach outperforms other models by a large margin in noise modeling and synthesis tasks. We also show that image denoisers trained on noisy images synthesized with our noise model outperforms those trained with noise from baselines models.