Repository: microsoft/superbenchmark

Issue:
V0.3.0 Release Plan
# Release Manager
@TobeyQin 

# Endgame
Code freeze: 9/1/2021
Bug Bash date: 9/2/2021
Release date: 9/17/2021

# Main Features
## SuperBench Framework

### SB Runner -- @abuccts 

- [x] MPI mode implementation
       PR: #146

### SB Benchmarks -- @guoshzhao 

- [x] Docker Base
       PR: #179 and #180

## Single-node Validation
### Micro-benchmarks -- @guoshzhao @yukirora 

1. - [x] Memory (Tool: Nvidia Bandwidth Test Tool) -- @yukirora ETA: 5/28/2021 
   PR: #114  
     | Metrics | Unit | Description | 
     |---|---|---| 
     | H2D_Mem_BW_\<GPU ID> | GB/s | host-to-GPU bandwidth for each GPU |
     | D2H_Mem_BW_\<GPU ID> | GB/s | GPU-to-host bandwidth  for each GPU |

2. - [ ] Device P2P Bandwidth (Tool: Nvidia p2pBandwidthLatencyTest Tool) -- Delayed

     | Metrics | Unit | Description | 
     |---|---|---|
     | P2P_BW_Max | GB/s | The maximum bandwidth in Bidirectional P2P=Enabled Bandwidth Matrix for all GPUs |
     | P2P_BW_Min | GB/s | The minimum bandwidth |
     | P2P_BW_Avg | GB/s | The average bandwidth |
   
3. - [x] IBLoopback (Tool: PerfTest – Standard RDMA Test Tool) -- @yukirora  ETA: 7/30/2021 
    PR: #112 and #129 
     | Metrics | Unit | Description | 
     |---|---|---| 
     | IB_Write | MB/s | The IB write loopback throughput with different message size |
     | IB_Read | MB/s | The IB read loopback throughput with different message size  |
     | IB_Send | MB/s | The IB send loopback throughput with different message size  |

4. - [x] NCCL (Tool: Nvidia NCCL Test) -- @yukirora  ETA: 7/30/2021
    PR: #113 and #128
     | Metrics | Unit | Description | 
     |---|---|---|
     | NCCL_AllReduce | GB/s | The NCCL AllReduce performance with different message size |
     | NCCL_AllGather | GB/s | The NCCL AllGather performance with different message size |
     | NCCL_broadcast | GB/s | The NCCL Broadcast performance with different message size |
     | NCCL_reduce | GB/s | The NCCL Reduce performance with different message size |
     | NCCL_reduce_scatter | GB/s | The NCCL ReduceScatter performance with different message size |

5. - [x] Disk (Tool: FIO – Standard Disk Performance Tool) -- @yzygitzh ETA: 7/30/2021
   PR: #127 and #132 and #161
     | Metrics | Unit | Description | 
     |---|---|---|
     | Seq_Read | MB/s | Sequential read performance |
     | Seq_Write | MB/s | Sequential write performance |
     | Rand_Read | MB/s | Random read performance |
     | Rand_Write | MB/s | Random write performance |
     | Seq_R/W_Read | MB/s | Read performance in sequential read/write, fixed measurement (read:write = 4:1)|
     | Seq_R/W_Write | MB/s | Write performance in sequential read/write (read:write = 4:1)|
     | Rand_R/W_Read | MB/s | Read performance in random read/write (read:write = 4:1)|
     | Rand_R/W_Write | MB/s | Write performance in random read/write (read:write = 4:1)|

6. - [x] H2D/D2H SM Transmission Bandwidth (Tool: MSR-A build) -- @yzygitzh ETA: 8/6/2021
   PR: #162 and #169    
     | Metrics | Unit | Description | 
     |---|---|---|
     | H2D_SM_BW_\<GPU ID>| GB/s | host-to-GPU bandwidth using GPU kernel for each GPU |
     | D2H_SM_BW_\<GPU ID> | GB/s | GPU-to-host bandwidth using GPU kernel for each GPU  |

### 
## Support AMD

### Docker Image Support  -- @guoshzhao ETA: 7/16/2021

- [x] ROCm 4.2 PyTorch 1.7 PR: #164
- [x] ROCm 4.0 PyTorch 1.7 PR: #164

### Micro Benchmarks
1. - [x] Kernel Launch (Tool: MSR-A build) -- @yukirora  ETA: 7/30/2021
  PR: #137 and #136  
      | Metrics | Unit | Description |
      |---|---|---|
      | Kernel_Launch_Event_Time | Time (ms) | Dispatch latency measured in GPU time using hipEventRecord() |
      |Kernel_Launch_Wall_Time| Time (ms) | Dispatch latency measured in CPU time |

2. - [x] RCCL (Tool: AMD RCCL Test) -- @yukirora ETA: 7/30/2021
  PR: #139 and #143  
     | Metrics | Unit | Description | 
     |---|---|---|
     | RCCL_AllReduce | GB/s | The RCCL AllReduce performance with different message size |
     | RCCL_AllGather | GB/s | The RCCL AllGather performance with different message size |
     | RCCL_broadcast | GB/s | The RCCL Broadcast performance with different message size |
     | RCCL_reduce | GB/s | The RCCL Reduce performance with different message size |
     | RCCL_reduce_scatter | GB/s | The RCCL ReduceScatter performance with different message size |

3. - [x] GEMM FLOPS (Tool: AMD rocblas-bench Tool) -- @yukirora ETA: 8/27/2021
     PR: #144 and #165
     | Metrics | Unit | Description |
     |---|---|---|
     | FP64 | GFLOPS | FP64 FLOPS without MatrixCore  |
     | FP32 | GFLOPS | FP32 FLOPS without MatrixCore  |
     | FP16 | GFLOPS | FP16 FLOPS without MatrixCore  |
     | FP32(MC) | GFLOPS | TF32 FLOPS with MatrixCore  |
     | FP16(MC) | GFLOPS | FP16 FLOPS with MatrixCore  |
     | BF16(MC) | GFLOPS | BF16 FLOPS with MatrixCore  |
     | INT8(MC) | GOPS | INT8 FLOPS with MatrixCore  |
     | INT4(MC) | GOPS | INT4 FLOPS with MatrixCore  |

4. - [x] Memory (Tool: HIP Bandwidth Test Tool) -- @yukirora ETA: 8/27/2021 
   PR: #159 and #153
     | Metrics | Unit | Description | 
     |---|---|---| 
     | H2D_Mem_BW_\<GPU ID> | GB/s | host-to-GPU bandwidth for each GPU |
     | D2H_Mem_BW_\<GPU ID> | GB/s | GPU-to-host bandwidth  for each GPU |

### E2E Benchmarks -- @guoshzhao ETA: 7/16/2021
1. - [x] CNN models -- User PyTorch TORCHVISION.MODELS sub-package
      - ResNet: ResNet-50, ResNet-101, ResNet-152
      - DenseNet: DenseNet-169, DenseNet-201 ​
      - VGG: VGG-11, VGG-13, VGG-16, VGG-19​
2. - [x] BERT -- Use huggingface Transformers
      - BERT
      - BERT LARGE
3. - [x] LSTM -- Use PyTorch TORCH.NN sub-package
4. - [x] GPT-2 --  Use huggingface Transformers

## Result Summary -- @cp5555

- [x] Generate a report to summarize the results -- @guoshzhao  ETA: 7/30/2021
       PR: #147, #149, and #157
- [ ] Support basic analysis feature (boxplot figure, outlier detection, etc.) 

## Bug Fix
- [x] VGG models failed on A100 GPU with batch_size=128 #115 
       PR: #134 
## Other Improvement

1. Contribution related -- @lynex 
   - [x] Contribute rule (#131)
   - [x] system information collection (#160)

2. Document -- @TobeyQin 
    - [x] Add release process doc (#130)
    - [x] Add design documents (#125)
    - [x] Add developer guide doc for coding style (#155)
    - [x] Add contribution rules (#131)
    - [x] Add docker image list (#154)
    - [x] Add initial validation results
    - [x] ~~Add metric reasoning doc -- @cp5555 @guoshzhao~~

3. Process monitor
    - [ ] Add Heart beat to monitor process health
    - [ ] Auto kill all processes on all nodes

4. Coding style -- @abuccts 
    - [x] Add vscode online

------------
## Backlogs

### Multi-Node Benchmarks
- Mellanox ClusterKit
- GPCNeT

### UI Design


Discussion:

Resolving patch:
diff --git a/superbench/__init__.py b/superbench/__init__.py
--- a/superbench/__init__.py
+++ b/superbench/__init__.py
@@ -6,5 +6,5 @@
 Provide hardware and software benchmarks for AI systems.
 """
 
-__version__ = '0.2.1'
+__version__ = '0.3.0'
 __author__ = 'Microsoft'


Test patch: