Collaborative Research: CyberTraining: Pilot: Research Workforce Development for Deep Learning Systems in Advanced GPU Cyberinfrastructure OAC-2306184/OAC-2330364 PI of OAC-2306184: Tong Shu, Department of Computer Science and Engineering, University of North Texas PI of OAC-2330364: Xin Liang, Department of Computer Science, University of Kentucky Training Program on Deep Learning Systems in Advanced GPU Cyberinfrastructure 2023 and 2024 https://dlgpu-workshop.github.io Training Syllabus Lecture Modules • For all students ➢Deep Neural Network Training and Inference on GPUs (Dr. Tony Luo) ➢Deep Learning Models I: Convolutional Neural Network and Physics- Informed Neural Network (Dr. Tony Luo) ➢Deep Learning Models II: Graph Neural Network and Transformer (Dr. Tong Shu) [New in 2024] ➢Resource-Aware Deep Learning Model Exploration on GPUs (Dr. Tong Shu) ➢Basic CUDA Programming (Dr. Tong Shu) ➢GPU Architectures and CUDA Programming (Dr. Tong Shu) • For CSE students ➢Advanced CUDA Programming (Dr. Xin Liang) ➢CUDA Programming for Deep Learning Applications (Dr. Xin Liang) ➢CUDA Libraries, Kernel Profiler, and Performance Analysis Tool (Dr. Iraklis Anagnostopoulos) ➢Deep Learning on TPUs (Dr. Iraklis Anagnostopoulos) [New in 2024] • For non-CSE students: ➢CNN-based Hydrological Connectivity Modeling (Dr. Ruopu Li) ➢PINN-Based Space Environment Nowcast (Dr. Daoru Han) Hands-on Exercise System https://github.com/SHUs-Lab/JupyterHubPlus • Configure JupyterHub for authentication and access of multiple users • Support CUDA C code block by modifying JupyterHub source code • Iterations with trainees and timely monitoring of progress on hands-on exercises Raw data (e.g., tabular data, images) Scientific/ Social Simulation/ Experiments DL-Driven Scientific Research Performance metrics (e.g., prediction accuracy, training time, resource efficiency) Search Space Neural Architecture | Hyper-Parameters | Resource Constraints GPUs of diverse capabilities (e.g., number of CUDA/tensor cores, peak TFLOPS, memory, bandwidth) PyTorch Model Exploration (NNI) Exploration Strategy Estimation Strategy Cross-model optimization Training control Search algorithms (e.g., evolutionary algorithm, reinforcement learning, bayesian optimization) Model generation with resource constraints Resource-Aware Neural Architecture Search CNN PINN Performance Modeling and Prediction DL inference DL training Nvidia GPUs Advanced CUDA Toolkit Architectures of State-of-the-Art GPUs PyTorch Convolutional-NN-based Hydrological Connectivity Modeling GPU-based DL Systems Advanced GPU CI DL App Benchmarks Geoscience Research Physics-Informed- NN-based Space Environment Nowcast Neural Network Intelligence (NNI) Resource-Aware DL Model Exploration Advanced Programming with Tensor Cores Latest Profiler: Nsight Compute, Nsight Systems DL Training and Inference on GPUs Libraries: cuBLAS, cuDNN, NCCL Interdisciplinary Projects • Project topics ➢Project 1: Accurate and time-efficient transformer inference on TPU architecture for drought classification (Summer 2024) ➢Project 2: Exploration of TPU architectures for optimized transformer performance in image detection of drainage crossings (Summer 2024) ➢Project 3: Accuracy-constrained efficiency optimization and GPU profiling of CNN inference for detecting drainage crossing locations (Summer 2023) ➢Project 4: Pareto optimization of CNN models via hardware-aware neural architecture search for drainage crossing classification on resource-limited devices (Summer 2023) ➢Project 5: Deep-learning model for optimization of an electrostatic Sieve for lunar regolith beneficiation (Summer 2024) ➢Project 6: Space environment modeling and uncertainty quantification with deep neural networks (Summer 2023) ➢Project 7: Reusing small pre-trained models for efficient large model training in natural language processing (Summer 2024) ➢Project 8: Real-time edge analytics for thermal infrared imagery and geological hazards with deep detection (Summer 2023) •