Reviewerly/exHarmony huggingface dataset:
DroneNet: Rescue Drone-View Object Detection

Recently, the research on drone-view object detection (DOD) has predominantly centered on efficiently identifying objects through cropping high-resolution images. However, it has overlooked the distinctive challenges posed by scale imbalance and a higher prevalence of small objects in drone images. In this paper, to address the challenges associated with the detection of drones (DODs), we introduce a specialized detector called DroneNet. Firstly, we propose a feature information enhancement module (FIEM) that effectively preserves object information and can be seamlessly integrated as a plug-and-play module into the backbone network. Then, we propose a split-concat feature pyramid network (SCFPN) that not only fuses feature information from different scales but also enables more comprehensive exploration of feature layers with many small objects. Finally, we develop a coarse to refine label assign (CRLA) strategy for small objects, which assigns labels from coarse- to fine-grained levels and ensures adequate training of small objects during the training process. In addition, to further promote the development of DOD, we introduce a new dataset named OUC-UAV-DET. Extensive experiments on VisDrone2021, UAVDT, and OUC-UAV-DET demonstrate that our proposed detector, DroneNet, exhibits significant improvements in handling challenging targets, outperforming state-of-the-art detectors.
FGBC: Flexible graph-based balanced classifier for class-imbalanced semi-supervised learning

Semi-supervised learning (SSL) has witnessed resounding success in many standard class-balanced benchmark datasets. However, real-world data often exhibit class-imbalanced distributions, which poses significant challenges for existing SSL algorithms. In general, fully supervised models trained on a class-imbalanced dataset are biased toward the majority classes, and this issue becomes more severe for class-imbalanced semi-supervised learning (CISSL) conditions. To address this issue, we put forward a novel CISSL framework dubbed FGBC by introducing a flexible graph-based balanced classifier with three innovations. Specifically, because the propagation of label information becomes difficult for tail classes, we propose a graph-based classifier head attached to the representation layer of the existing SSL framework for efficient pseudo-label propagation. Then, by considering that the learning status of different classes in CISSL may vary, we introduce a flexible threshold adjustment in pseudo-labeling to further select balanced samples to participate in training. Furthermore, to alleviate the risk of overfitting tail classes, we devised a class-aware feature MixUp (CFM) augmentation algorithm, which can further enhance the features of each class by considering their class sizes. Experimental results demonstrate that FGBC achieves state-of-the-art performance on datasets from CIFAR-10/100, SVHN and Small ImageNet-127 under various levels of CISSL conditions.
AirTrack: Onboard Deep Learning Framework for Long-Range Aircraft Detection and Tracking

Detect-and-Avoid (DAA) capabilities are critical for safe operations of unmanned aircraft systems (UAS). This paper introduces, AirTrack, a real-time vision-only detect and tracking framework that respects the size, weight, and power (SWaP) constraints of sUAS systems. Given the low Signal-to-Noise ratios (SNR) of far away aircraft, we propose using full resolution images in a deep learning framework that aligns successive images to remove ego-motion. The aligned images are then used downstream in cascaded primary and secondary classifiers to improve detection and tracking performance on multiple metrics. We show that AirTrack outperforms state-of-the art baselines on the Amazon Airborne Object Tracking (AOT) Dataset. Multiple real world flight tests with a Cessna 182 interacting with general aviation traffic and additional near-collision flight tests with a Bell helicopter flying towards a UAS in a controlled setting showcase that the proposed approach satisfies the newly introduced ASTM F3442/F3442M standard for DAA. Empirical evaluations show that our system has a probability of track of more than 95% up to a range of 700m. [Video] <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Video: https://youtu.be/bMw5nUGL5GQ
Navigation with polytopes and B-spline path planner

This paper firstly presents our optimal path planning algorithm within a <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$2\mathrm{D}$</tex> non-convex, polytopic region defined as a sequence of connected convex polytopes. The path is a B-spline curve but being parametrized with its equivalent Bézier representation. By doing this, the local convexity bound of each curve's interval is significantly tighter. Thus, it allows many more possibilities for constraining the entire curve to remain inside the region by using only linear constraints on the control points of the curve. We further guarantee the existence of the valid path by pointing out an algebraic solution. We integrate the algorithm, together with our previously published results, into the Navigation with polytopes toolbox which can be used as a global path planner, compatible with ROS navigation tools. It provides a framework for constructing a polytope map from a standard occupancy gridmap, searching for an appropriate sequence of connected polytopes and finally, planning a minimal-length path with different options on B-spline or Bézier parametrizations. The validation and comparison with existing methods are done using gridmaps collected under Gazebo simulations and real experiments.
NeRF2Real: Sim2real Transfer of Vision-guided Bipedal Motion Skills using Neural Radiance Fields

We present a system for applying sim2real approaches to “in the wild” scenes with realistic visuals, and to policies which rely on active perception using RGB cameras. Given a short video of a static scene collected using a generic phone, we learn the scene's contact geometry and a function for novel view synthesis using a Neural Radiance Field (NeRF). We augment the NeRF rendering of the static scene by overlaying the rendering of other dynamic objects (e.g. the robot's own body, a ball). A simulation is then created using the rendering engine in a physics simulator which computes contact dynamics from the static scene geometry (estimated from the NeRF vol-ume density) and the dynamic objects' geometry and physical properties (assumed known). We demonstrate that we can use this simulation to learn vision-based whole body navigation and ball pushing policies for a 20 degree-of-freedom humanoid robot with an actuated head-mounted RGB camera, and we successfully transfer these policies to a real robot.
ADAPT: Action-aware Driving Caption Transformer

End-to-end autonomous driving has great potential in the transportation industry. However, the lack of transparency and interpretability of the automatic decision-making process hinders its industrial adoption in practice. There have been some early attempts to use attention maps or cost volume for better model explainability which is difficult for ordinary passengers to understand. To bridge the gap, we propose an end-to-end transformer-based architecture, ADAPT (Action-aware Driving cAPtion Transformer), which provides user-friendly natural language narrations and reasoning for each decision making step of autonomous vehicular control and action. ADAPT jointly trains both the driving caption task and the vehicular control prediction task, through a shared video representation. Experiments on BDD-X (Berkeley DeepDrive eXplanation) dataset demonstrate state-of-the-art performance of the ADAPT framework on both automatic metrics and human evaluation. To illustrate the feasibility of the proposed framework in real-world applications, we build a novel deployable system that takes raw car videos as input and outputs the action narrations and reasoning in real time. The code, models and data are available at https://github.com/jxbbb/ADAPT.
A Meta-Summary of Challenges in Building Products with ML Components – Collecting Experiences from 4758+ Practitioners

Incorporating machine learning (ML) components into software products raises new software-engineering challenges and exacerbates existing ones. Many researchers have invested significant effort in understanding the challenges of industry practitioners working on building products with ML components, through interviews and surveys with practitioners. With the intention to aggregate and present their collective findings, we conduct a meta-summary study: We collect 50 relevant papers that together interacted with over 4758 practitioners using guidelines for systematic literature reviews. We then collected, grouped, and organized the over 500 mentions of challenges within those papers. We highlight the most commonly reported challenges and hope this meta-summary will be a useful resource for the research community to prioritize research and education in this field.
Exploring a radically new exponential Retinex model for multi-task environments

The Retinex Theory (RT) and its adaptations have gained significant popularity in the field of image processing. Nevertheless, traditional Retinex algorithms are generally customized to specific tasks. Besides, their use of logarithmic transformation (LT) to convert the multiplicative model to an additive one often results in the loss of texture information in the reflectance layer. In contrast to conventional methods, our approach involves direct decomposition of the observed image. This approach circumvents the necessity of intermediate transformations, thereby preserving essential texture features. In this study, we introduce a weight-aware ℓ1-ℓ2 technique based on the assumption that the reflectance layer is discontinuous and the illumination layer is spatially smooth. To preserve texture and structural information in the illumination layer, we introduce a weight-aware illumination component coefficient, ℓ1-norm, and estimate the reflectance component using ℓ2-norm. By utilizing weight-aware coefficients, the proposed technique is highly effective in addressing the issue of texture loss in the reflectance layer. Additionally, we employ ℓ2-norm to extract accurate information from the reflectance layer and implement a bright channel prior to prevent ambiguity during the decomposition process. We utilize an alternating minimization approach to obtain the optimal objective function solution and modify the illumination layer using gamma and non-linear stretching algorithms. Our proposed technique not only tackles the problem of texture duplication but also improves the quality of low-light images, and can be seamlessly integrated with various image and vision-based tasks. Our evaluation of eight benchmark datasets using 15 quality metrics, along with a variety of 22 conventional and modern algorithms, shows that the proposed algorithm is capable of delivering competitive qualitative and quantitative results without compromising its flexibility and scalability. Besides, the proposed model is evaluated on retinal images, and the results demonstrate a substantial improvement in the accuracy of learning-based models.
Next-power: Next-generation framework for secure and sustainable energy trading in the metaverse

The rapid emergence of the Metaverse as a leading virtual digital technology is driving innovation in power generation. The increasing demand for sustainable energy and the rise of the Metaverse has provided an opportunity to develop a secure and sustainable energy trading system using Smart Grid (SG), Virtual Power Plants (VPP), Digital Twins (DT), and Blockchain. Integrating VPP with SG can optimize the utilization of distributed energy resources and improve the stability and reliability of the energy supply in the metaverse. Meanwhile, the use of DT can facilitate the efficient management of VPPs by simulating their behavior and performance to identify potential issues and test different scenarios for improvement. Blockchain is a rising technology that enables decentralized and trustworthy applications, making it a promising tool for creating decentralized energy trading systems in the Metaverse. In this context, we propose Next-Power, a next-generation framework for secure and sustainable energy trading in the Metaverse. Next-Power consists of: (1) an automated blockchain-based fair payment of energy traded between consumers and prosumers in the Metaverse, it uses Ethereum’s smart contracts to determine the price at which energy is traded using a cooperative game theory (i.e., reverse auction); (2) an access token-based blockchain system that ensures a secure mutual authentication between consumers and prosumers; and (3) a lightweight on-chain collaborative reputation scheme that keeps track of prosumers reputation as well as consumers credibility in the Metaverse in a fully decentralized manner. The system of Next-Power is implemented, tested, and deployed on the Ethereum official test network. Next-Power is a promising framework for the future of energy trading in the Metaverse, as demonstrated by its successful experimental results in terms of flexibility, efficiency, security, and cost-effectiveness.
Philosophical underpinnings of digital citizenship through a postdigital lens: Implications for teacher educators’ professional digital competence

Abstract Embedded in society, digital infrastructure has changed citizens’ lives. Young people therefore need to develop digital competence and digital citizenship, and schools have an important role in this regard. To prepare new schoolteachers for this role, teacher educators (TEDs) need professional digital competence (PDC) that includes knowledge, competences, and a conceptual understanding to teach teaching for digital citizenship. In light of the limited body of research on theorizing digital citizenship in relation to TEDs’ PDC, this paper critically analyzes three conceptualizations of digital citizenship. Being potentially normative and part of the latest phase of development in the field, these conceptualizations could shape TEDs’ PDC and practice. In a qualitative content analysis of the selected conceptualizations, this paper uses a postdigital lens to bring into focus and critically analyze aspects of philosophical underpinnings related to socio-technical relations. The results show that conceptualizations of digital citizenship convey different understandings of human–technology relations and the knowledge and competences necessary to exercise digital citizenship. These differences have far-reaching implications for TEDs’ PDC in ways that could impact students’ opportunities to develop digital competence and digital citizenship. Therefore, TEDs’ PDC needs to include a critical understanding of digital citizenship, and the post-pandemic juncture of “new normal” provides opportunities to rethink and reframe PDC. To this end, a postdigital lens can shift the focus to how PDC is contingent on the shifting entanglements in which pedagogical activities are situated and orchestrated, and how these relate to broader issues of injustice in society.
Digital transformation in municipalities for the planning, delivery, use and management of infrastructure assets: Strategic and organizational framework

As the acceleration of technological development in the built asset industry brings on waves of digital transformation (DT), traditional ways of doing and organizing are being disrupted, especially on the part of large public owners such as municipalities. For these owners, these waves of transformation require constant adaptation as they compete with existing initiatives and embedded legacy practices. This paper presents the results of the second part of a longitudinal research project aimed at framing digital transformation within municipalities to improve urban infrastructure lifecycles. More specifically, the paper presents the results of work undertaken to operationalize, extend and further validate the digital transformation framework that has been developed in part 1 and which is presented elsewhere. The theoretical framework acts as a guide and analysis tool for the digital transformation of municipalities and aims to help them reduce and/or eliminate the barriers and challenges in this digital transformation. To do so, the results from a survey conducted within 44 municipalities and interviews conducted with 13 municipalities of different sizes are presented and discussed through the theoretical framework. The results show that data and information management remain the key issues, especially in a siloed organizational context such as those found within municipalities. Moreover, a significant amount of organizations remain unaware of how to approach digital transformation which in turn leads to disinterest or disengagement in digital transformation, which results in localized or fragmented initiatives. This in turn can cause delays in implementing transformational initiatives and contributes to maintaining a low level of digital maturity. The study also highlights the critical lack of human resources, expertise and appropriate training to support digital transformation.
GNM: A General Navigation Model to Drive Any Robot

Learning provides a powerful tool for vision-based navigation, but the capabilities of learning-based policies are constrained by limited training data. If we could combine data from all available sources, including multiple kinds of robots, we could train more powerful navigation models. In this paper, we study how a general goal-conditioned model for vision-based navigation can be trained on data obtained from many distinct but structurally similar robots, and enable broad generalization across environments and embodiments. We analyze the necessary design decisions for effective data sharing across robots, including the use of temporal context and standardized action spaces, and demonstrate that an omnipolicy trained from heterogeneous datasets outperforms policies trained on any single dataset. We curate 60 hours of navigation trajectories from 6 distinct robots, and deploy the trained GNM on a range of new robots, including an underactuated quadrotor. We find that training on diverse data leads to robustness against degradation in sensing and actuation. Using a pre-trained navigation model with broad generalization capabilities can bootstrap applications on novel robots going forward, and we hope that the GNM represents a step in that direction. For more information on the datasets, code, and videos, please check out our project page <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> sites.google.com/view/drive-any-robot.
ATTACH Dataset: Annotated Two-Handed Assembly Actions for Human Action Understanding

With the emergence of collaborative robots (cobots), human-robot collaboration in industrial manufacturing is coming into focus. For a cobot to act autonomously and as an assistant, it must understand human actions during assembly. To effectively train models for this task, a dataset containing suitable assembly actions in a realistic setting is cru-cial. For this purpose, we present the ATTACH dataset, which contains 51.6 hours of assembly with 95.2k annotated fine-grained actions monitored by three cameras, which represent potential viewpoints of a cobot. Since in an assembly context workers tend to perform different actions simultaneously with their two hands, we annotated the performed actions for each hand separately. Therefore, in the ATTACH dataset, more than 68% of annotations overlap with other annotations, which is many times more than in related datasets, typically featuring more simplistic assembly tasks. For better generalization with respect to the background of the working area, we did not only record color and depth images, but also used the Azure Kinect body tracking SDK for estimating 3D skeletons of the worker. To create a first baseline, we report the performance of state-of-the-art methods for action recognition as well as action detection on video and skeleton-sequence inputs. The dataset is available at https://www.tu-ilmenau.de/neurob/data-sets-code/attach-dataset.
Hierarchical Policy Blending as Inference for Reactive Robot Control

Motion generation in cluttered, dense, and dynamic environments is a central topic in robotics, rendered as a multi-objective decision-making problem. Current approaches trade-off between safety and performance. On the one hand, reactive policies guarantee a fast response to environmental changes at the risk of suboptimal behavior. On the other hand, planning-based motion generation provides feasible trajectories, but the high computational cost may limit the control frequency and, thus, safety. To combine the benefits of reactive policies and planning, we propose a hierarchical motion generation method. Moreover, we employ probabilistic inference methods to formalize the hierarchical model and stochastic optimization. We realize this approach as a weighted product of stochastic, reactive expert policies, where planning is used to adaptively compute the optimal weights over the task horizon. This stochastic optimization avoids local optima and proposes feasible reactive plans that find paths in cluttered and dense environments. Our extensive experimental study in planar navigation and 7DoF manipulation shows that our proposed hierarchical motion generation method outperforms both myopic reactive controllers and online re-planning methods. Additional material available at https://sites.google.com/view/hipbi.
Exploiting Trust for Resilient Hypothesis Testing with Malicious Robots

We develop a resilient binary hypothesis testing frame-work for decision making in adversarial multi-robot crowdsensing tasks. This framework exploits stochastic trust observations between robots to arrive at tractable, resilient decision making at a centralized Fusion Center (FC) even when i) there exist malicious robots in the network and their number may be larger than the number of legitimate robots, and ii) the FC uses one-shot noisy measurements from all robots. We derive two algorithms to achieve this. The first is the Two Stage Approach (2SA) that estimates the legitimacy of robots based on received trust observations, and provably minimizes the probability of detection error in the worst-case malicious attack. Here, the proportion of malicious robots is known but arbitrary. For the case of an unknown proportion of malicious robots, we develop the Adversarial Generalized Likelihood Ratio Test (A-GLRT) that uses both the reported robot measurements and trust observations to estimate the trustworthiness of robots, their reporting strategy, and the correct hypothesis simultaneously. We exploit special problem structure to show that this approach remains computationally tractable despite several unknown problem parameters. We deploy both algorithms in a hardware experiment where a group of robots conducts crowdsensing of traffic conditions on a mock-up road network similar in spirit to Google Maps, subject to a Sybil attack. We extract the trust observations for each robot from actual communication signals which provide statistical information on the uniqueness of the sender. We show that even when the malicious robots are in the majority, the FC can reduce the probability of detection error to 30.5% and 29% for the 2SA and the A-GLRT respectively.