Reviewerly/exHarmony huggingface dataset:
Multimodal learning with graphs

Artificial intelligence for graphs has achieved remarkable success in modelling complex systems, ranging from dynamic networks in biology to interacting particle systems in physics. However, the increasingly heterogeneous graph datasets call for multimodal methods that can combine different inductive biases — assumptions that algorithms use to make predictions for inputs they have not encountered during training. Learning on multimodal datasets is challenging because the inductive biases can vary by data modality and graphs might not be explicitly given in the input. To address these challenges, graph artificial intelligence methods combine different modalities while leveraging cross-modal dependencies through geometric relationships. Diverse datasets are combined using graphs and fed into sophisticated multimodal architectures, specified as image-intensive, knowledge-grounded and language-intensive models. Using this categorization, we introduce a blueprint for multimodal graph learning, use it to study existing methods and provide guidelines to design new models. One of the main advances in deep learning in the past five years has been graph representation learning, which enabled applications to problems with underlying geometric relationships. Increasingly, such problems involve multiple data modalities and, examining over 160 studies in this area, Ektefaie et al. propose a general framework for multimodal graph learning for image-intensive, knowledge-grounded and language-intensive problems.
A Survey on Deep Reinforcement Learning Algorithms for Robotic Manipulation

Robotic manipulation challenges, such as grasping and object manipulation, have been tackled successfully with the help of deep reinforcement learning systems. We give an overview of the recent advances in deep reinforcement learning algorithms for robotic manipulation tasks in this review. We begin by outlining the fundamental ideas of reinforcement learning and the parts of a reinforcement learning system. The many deep reinforcement learning algorithms, such as value-based methods, policy-based methods, and actor-critic approaches, that have been suggested for robotic manipulation tasks are then covered. We also examine the numerous issues that have arisen when applying these algorithms to robotics tasks, as well as the various solutions that have been put forth to deal with these issues. Finally, we highlight several unsolved research issues and talk about possible future directions for the subject.
Base and Meta: A New Perspective on Few-Shot Segmentation

Despite the progress made by few-shot segmentation (FSS) in low-data regimes, the generalization capability of most previous works could be fragile when countering hard query samples with seen-class objects. This paper proposes a fresh and powerful scheme to tackle such an intractable bias problem, dubbed base and meta (BAM). Concretely, we apply an auxiliary branch (base learner) to the conventional FSS framework (meta learner) to explicitly identify base-class objects, i.e., the regions that do not need to be segmented. Then, the coarse results output by these two learners in parallel are adaptively integrated to derive accurate segmentation predictions. Considering the sensitivity of meta learner, we further introduce adjustment factors to estimate the scene differences between support and query image pairs from both style and appearance perspectives, so as to facilitate the model ensemble forecasting. The remarkable performance gains on standard benchmarks (PASCAL-5 i, COCO-20 i, and FSS-1000) manifest the effectiveness, and surprisingly, our versatile scheme sets new state-of-the-arts even with two plain learners. Furthermore, in light of its unique nature, we also discuss several more practical but challenging extensions, including generalized FSS, 3D point cloud FSS, class-agnostic FSS, cross-domain FSS, weak-label FSS, and zero-shot segmentation. Our source code is available at https://github.com/chunbolang/BAM.
Tungsten and tin deposits in South China: Temporal and spatial distribution, metallogenic models and prospecting directions

South China is one of the most important tungsten-tin metallogenic areas in the world. Based on the up-to-date geological and chronological data of more than 130 deposits of economic significance, this paper preliminarily summarized the main types and temporal-spatial distribution characteristics of the large-scale tungsten and tin mineralization related to granite in South China. The statistical results show that skarn and quartz-vein types are the most important tungsten metallogenic styles, while skarn and cassiterite-sulfide types are the most important tin metallogenic styles in South China. Tungsten and tin mineralization related to granite in South China are multi-aged, but the largest scale of mineralization is developed within the Yanshanian period. Tungsten mineralization, the majority of which represented by the Nanling and Ganbei metallogenic belts, mainly formed from the Late Jurassic to Early Cretaceous (160 ∼ 120 Ma), while the most important tin mineralization formed in the Late Cretaceous (110 ∼ 80 Ma, peaking at 90 ∼ 80 Ma) and Late Jurassic (160 ∼ 150 Ma) as represented by the Youjiang Basin and Nanling metallogenic belt, respectively. The combined occurrence of different mineralization styles or metal assemblages in individual deposit and sometimes within the same orefield is the most striking feature of granite-related tungsten-tin deposits in South China, and a comprehensive understanding of their patterns are of great significance to guide ore prospecting. To address this issue, this paper proposed several combined metallogenic models and prospecting directions for tungsten-tin mineralization in South China based on recent researches and exploration progresses from typical examples including the Yaogangxian, Chuankou, Maoping, Shizhuyuan, Dachang and Gejiu deposits or orefields. In addition, conventional petrological and mineralogical characteristics that differentiates tungsten and tin granites are summarized, and on this basis the chemical composition of both melt inclusion and early stage fluid inclusion from granitic magmatic-hydrothermal systems were tested as new index to evaluate the tungsten and tin mineralization potential of granitic system.
Exploring the Potential of Metaverse for Higher Education: Opportunities, Challenges, and Implications

Introduction: The Metaverse, a virtual space where users can engage with each other in a variety of experiences, has gained significant attention in recent years. With the advent of advanced technologies like Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), the possibilities of the Metaverse have expanded to create new opportunities for learning and education. Objective: This research article aims to explore the potential of the Metaverse for higher education, including its opportunities, challenges, and implications. Methods: The article provides a comprehensive overview of the Metaverse, its history, and its key components. It then examines the potential of the Metaverse for higher education by reviewing relevant literature and analyzing the benefits that it can offer. These benefits include increased accessibility, flexibility, and interactivity, as well as the potential for immersive and personalized learning experiences. Results: The article also identifies the challenges and limitations of the Metaverse, such as technical limitations, privacy and security concerns, and the need for digital literacy skills. It further discusses the implications of using the Metaverse for higher education, including its impact on teaching and learning, curriculum design, and student engagement. Conclusions: Finally, the article proposes some recommendations for educators and institutions interested in exploring the use of the Metaverse for higher education. These recommendations include developing a clear understanding of the educational goals, selecting appropriate technologies and platforms, and providing training and support for both educators and students.
A framework for AI-powered service innovation capability: Review and agenda for future research

Artificial intelligence (AI)-powered service innovation (e.g., OpenAI's ChatGPT, Google's Bard and Microsoft's Sydney) has become one of the most significant determinants of firms' success in the Fourth Industrial Revolution. However, extant studies on this topic show that research studies hitherto have been ad-hoc, lacking a conceptual framework for the strategic management of AI-powered service innovation capability in dynamic markets. Thus, this study synthesises the current body of knowledge, proposes a framework, and develops an agenda to advance our knowledge. The findings reveal: (1) AI-market capability relates to customer orientation, industry orientation, and cross-functional orientation; (2) AI-infrastructure capability relates to data, business models, and ecosystem; and (3) AI-management capability relates to AI-orientation, organisational learning, and AI ethics which are crucial determinants of forming AI-powered service innovation capability. These capabilities for the strategic management of AI-powered service innovations play an essential role in achieving organizational agility and competitive advantage.
WaveNet: Wavelet Network With Knowledge Distillation for RGB-T Salient Object Detection

In recent years, various neural network architectures for computer vision have been devised, such as the visual transformer and multilayer perceptron (MLP). A transformer based on an attention mechanism can outperform a traditional convolutional neural network. Compared with the convolutional neural network and transformer, the MLP introduces less inductive bias and achieves stronger generalization. In addition, a transformer shows an exponential increase in the inference, training, and debugging times. Considering a wave function representation, we propose the WaveNet architecture that adopts a novel vision task-oriented wavelet-based MLP for feature extraction to perform salient object detection in RGB (red-green-blue)-thermal infrared images. In addition, we apply knowledge distillation to a transformer as an advanced teacher network to acquire rich semantic and geometric information and guide WaveNet learning with this information. Following the shortest-path concept, we adopt the Kullback-Leibler distance as a regularization term for the RGB features to be as similar to the thermal infrared features as possible. The discrete wavelet transform allows for the examination of frequency-domain features in a local time domain and time-domain features in a local frequency domain. We apply this representation ability to perform cross-modality feature fusion. Specifically, we introduce a progressively cascaded sine-cosine module for cross-layer feature fusion and use low-level features to obtain clear boundaries of salient objects through the MLP. Results from extensive experiments indicate that the proposed WaveNet achieves impressive performance on benchmark RGB-thermal infrared datasets. The results and code are publicly available at https://github.com/nowander/WaveNet.
GTR-GA: Harnessing the power of graph-based neural networks and genetic algorithms for text augmentation

Text augmentation is a popular technique in natural language processing (NLP) that has been shown to improve the performance of various downstream tasks. The goal of text augmentation is to generate additional training data from existing data, thereby increasing the amount of data available for training machine learning models. In this paper, we propose a novel approach to text augmentation that combines the power of graph-based neural networks and genetic algorithms. Our proposed scheme, called GTR-GA, is designed to generate diverse and high-quality augmented text data by exploring the high-dimensional feature space of text data using graph-based neural networks and genetic algorithms. Our approach utilizes a graph attention network (GAT) based model, called HetGAPN, to learn node representations in a heterogeneous graph representing text data. We introduce the concept of node aggregation and bidirectional attention in HetGAPN to better capture the relationships between different features of the input text data. We then use a genetic algorithm to generate a diverse set of candidate embeddings that explore different parts of the input space. Our approach uses perplexity as the objective function to evaluate the quality of the augmented text data generated by each candidate embedding. The most fit candidate embeddings are selected as parents, and crossover and mutation operators are used to generate a new population of candidate embeddings. Our experimental results show that GTR-GA is capable of generating high-quality augmented text data that improves the performance of various downstream NLP tasks, such as sentiment analysis and text classification. The proposed scheme can be applied to a wide range of NLP tasks and can help overcome the data scarcity problem that is often encountered in NLP.
Code as Policies: Language Model Programs for Embodied Control

Large language models (LLMs) trained on code-completion have been shown to be capable of synthesizing simple Python programs from docstrings [1]. We find that these code-writing LLMs can be re-purposed to write robot policy code, given natural language commands. Specifically, policy code can express functions or feedback loops that process perception outputs (e.g., from object detectors [2], [3]) and parameterize control primitive APIs. When provided as input several example language commands (formatted as comments) followed by corresponding policy code (via few-shot prompting), LLMs can take in new commands and autonomously re-compose API calls to generate new policy code respectively. By chaining classic logic structures and referencing third-party libraries (e.g., NumPy, Shapely) to perform arithmetic, LLMs used in this way can write robot policies that (i) exhibit spatial-geometric reasoning, (ii) generalize to new instructions, and (iii) prescribe precise values (e.g., velocities) to ambiguous descriptions (‘faster’) depending on context (i.e., behavioral commonsense). This paper presents Code as Policies: a robot-centric formulation of language model generated programs (LMPs) that can represent reactive policies (e.g., impedance controllers), as well as waypoint-based policies (vision-based pick and place, trajectory-based control), demonstrated across multiple real robot platforms. Central to our approach is prompting hierarchical code-gen (recursively defining undefined functions), which can write more complex code and also improves state-of-the-art to solve 39.8% of problems on the HumanEval [1] benchmark. Code and videos are available at https://code-as-policies.github.io
Short-term photovoltaic power forecasting using meta-learning and numerical weather prediction independent Long Short-Term Memory models

Short-term photovoltaic (PV) power forecasting is essential for integrating renewable energy sources into the grid as it provides accurate and timely information on the expected output of PV systems. Deep learning (DL) networks have shown promising results in this area, but depending on the weather conditions and the particularities of each PV system, different DL architectures may perform best. This paper proposes a meta-learning method to improve one-hour-ahead deterministic forecasts of PV systems by dynamically blending the base forecasts of multiple DL models to learn under what conditions each model performs best. Four base models of different long short-term memory architectures are used to produce PV production forecasts without using numerical weather predictions, with the objective to enhance the generalizability of the proposed solution. The accuracy of the meta-learner is evaluated using three rooftop PV systems in Lisbon, Portugal. Results indicate that different base models perform best at different PV plants, and meta-learning can improve accuracy by up to 5% over the most accurate base model per plant and up to 4.5% over the equal-weighted combination of the base forecasts. These improvements are statistically significant and even larger during peak production hours.
KnowleNet: Knowledge fusion network for multimodal sarcasm detection

Sarcasm is a form of communication often used to express contempt or ridicule, where the speaker conveys a message opposite to their true meaning, typically intending to mock or belittle a specific target. Sarcasm detection has gained great attention in the field of natural language processing due to the fact that sarcasm is widespread on social media and difficult to detect for machines. While early efforts in sarcasm detection solely relied on textual data, the abundance of multimodal data on social media is also non-negligible. Recent research has focused on multimodal sarcasm detection, where attention mechanisms and graph neural networks were commonly used to identify relevant information in both image and text data. However, these methods may overlook the importance of prior knowledge and cross-modal semantic contrast, which are crucial factors for human sarcasm detection. In this paper, we propose a novel model named KnowleNet that leverages the ConceptNet knowledge base to incorporate prior knowledge and determine image–text relatedness through sample-level and word-level cross-modal semantic similarity detection. Contrastive learning is also introduced to improve the spatial distribution of sarcastic (positive) and non-sarcastic (negative) samples. The proposed model achieves state-of-the-art performance on publicly available benchmark datasets.
Recovering 3D Human Mesh From Monocular Images: A Survey

Estimating human pose and shape from monocular images is a long-standing problem in computer vision. Since the release of statistical body models, 3D human mesh recovery has been drawing broader attention. With the same goal of obtaining well-aligned and physically plausible mesh results, two paradigms have been developed to overcome challenges in the 2D-to-3D lifting process: i) an optimization-based paradigm, where different data terms and regularization terms are exploited as optimization objectives; and ii) a regression-based paradigm, where deep learning techniques are embraced to solve the problem in an end-to-end fashion. Meanwhile, continuous efforts are devoted to improving the quality of 3D mesh labels for a wide range of datasets. Though remarkable progress has been achieved in the past decade, the task is still challenging due to flexible body motions, diverse appearances, complex environments, and insufficient in-the-wild annotations. To the best of our knowledge, this is the first survey that focuses on the task of monocular 3D human mesh recovery. We start with the introduction of body models and then elaborate recovery frameworks and training objectives by providing in-depth analyses of their strengths and weaknesses. We also summarize datasets, evaluation metrics, and benchmark results. Open issues and future directions are discussed in the end, hoping to motivate researchers and facilitate their research in this area. A regularly updated project page can be found at https://github.com/tinatiansjz/hmr-survey.
MobileOne: An Improved One millisecond Mobile Backbone

Efficient neural network backbones for mobile devices are often optimized for metrics such as FLOPs or parameter count. However, these metrics may not correlate well with latency of the network when deployed on a mobile device. Therefore, we perform extensive analysis of different metrics by deploying several mobile-friendly networks on a mobile device. We identify and analyze architectural and optimization bottlenecks in recent efficient neural networks and provide ways to mitigate these bottlenecks. To this end, we design an efficient backbone MobileOne, with variants achieving an inference time under 1 ms on an iPhone12 with 75.9% top-1 accuracy on ImageNet. We show that MobileOne achieves state-of-the-art performance within the efficient architectures while being many times faster on mobile. Our best model obtains similar performance on ImageNet as MobileFormer while being 38×faster. Our model obtains 2.3% better top-1 accuracy on ImageNet than EfficientNet at similar latency. Furthermore, we show that our model generalizes to multiple tasks - image classification, object detection, and semantic segmentation with significant improvements in latency and accuracy as compared to existing efficient architectures when deployed on a mobile device. Code and models are available at https://github.com/apple/ml-mobileone
N-Gram in Swin Transformers for Efficient Lightweight Image Super-Resolution

While some studies have proven that Swin Transformer (Swin) with window self-attention (WSA) is suitable for single image super-resolution (SR), the plain WSA ignores the broad regions when reconstructing high-resolution images due to a limited receptive field. In addition, many deep learning SR methods suffer from intensive computations. To address these problems, we introduce the N-Gram context to the low-level vision with Transformers for the first time. We define N-Gram as neighboring local windows in Swin, which differs from text analysis that views N-Gram as consecutive characters or words. N-Grams interact with each other by sliding-WSA, expanding the regions seen to restore degraded pixels. Using the N-Gram context, we propose NGswin, an efficient SR network with SCDP bottleneck taking multi-scale outputs of the hierarchical encoder. Experimental results show that NGswin achieves competitive performance while maintaining an efficient structure when compared with previous leading methods. Moreover, we also improve other Swin-based SR methods with the N-Gram context, thereby building an enhanced model: SwinIR-NG. Our improved SwinIR-NG out-performs the current best lightweight SR approaches and establishes state-of-the-art results. Codes are available at https://github.com/rami0205/NGramSwin.
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers

Scene understanding based on image segmentation is a crucial component of autonomous vehicles. Pixel-wise semantic segmentation of RGB images can be advanced by exploiting complementary features from the supplementary modality ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">${X}$ </tex-math></inline-formula> -modality). However, covering a wide variety of sensors with a modality-agnostic model remains an unresolved problem due to variations in sensor characteristics among different modalities. Unlike previous modality-specific methods, in this work, we propose a unified fusion framework, CMX, for RGB-X semantic segmentation. To generalize well across different modalities, that often include supplements as well as uncertainties, a unified cross-modal interaction is crucial for modality fusion. Specifically, we design a Cross-Modal Feature Rectification Module (CM-FRM) to calibrate bi-modal features by leveraging the features from one modality to rectify the features of the other modality. With rectified feature pairs, we deploy a Feature Fusion Module (FFM) to perform sufficient exchange of long-range contexts before mixing. To verify CMX, for the first time, we unify five modalities complementary to RGB, i.e., depth, thermal, polarization, event, and LiDAR. Extensive experiments show that CMX generalizes well to diverse multi-modal fusion, achieving state-of-the-art performances on five RGB-Depth benchmarks, as well as RGB-Thermal, RGB-Polarization, and RGB-LiDAR datasets. Besides, to investigate the generalizability to dense-sparse data fusion, we establish an RGB-Event semantic segmentation benchmark based on the EventScape dataset, on which CMX sets the new state-of-the-art. The source code of CMX is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/huaaaliu/RGBX_Semantic_</uri> Segmentation.