Repository: volcengine/verl
Pull request #1421: [utils] Enrich features to a few `verl` utilities
Author: login: hongpeng-guo
Labels: name: status: need review
Created: 2025-05-06T18:58:39Z | Merged: 2025-05-09T23:50:59Z

### Checklist Before Starting

- [x] Search for similar PR(s).

### What does this PR do?

1. In `verl/utils/fs.py`: adding a `always_recopy` kwarg to `copy_to_local` function, so that the file copied will always be triggered regardless of the local cache, which could be useful in certain cases.
2. In `verl/utils/torch_funcitonal`.py: adding two utilities `distributed_mean_max_min_std` and `distributed_masked_mean` to distributed torch tensor stats collection.

### Test

1. Added `tests/verl/utils/test_fs.py` and ``tests/verl/utils/test_torch_functional.py` to unit test the corresponding utility functions.
2. Changing `verl_unit_test` to run a [8*L20] to run torch tests, which uses cuda and nccl.


### Checklist Before Submitting

- [x] Read the [Contribute Guide](https://github.com/volcengine/verl?tab=readme-ov-file#contribution-guide).
- [x] Apply [pre-commit checks](https://github.com/volcengine/verl?tab=readme-ov-file#code-linting-and-formatting).
- [ ] Add `[BREAKING]` to the PR title if it breaks any API.
- [ ] Update the documentation about your changes in the [docs](https://github.com/volcengine/verl/tree/main/docs).
- [x] Add CI test(s) if neccessary.



Diff:
diff --git a/.github/workflows/verl_unit_test.yml b/.github/workflows/verl_unit_test.yml
index 7305a89076e..3583292082e 100644
--- a/.github/workflows/verl_unit_test.yml
+++ b/.github/workflows/verl_unit_test.yml
@@ -48,4 +48,4 @@ jobs:
       - name: Running utils tests
         run: |
           cd tests/verl/utils
-          pytest -s -x --ignore=dataset/ .
+          pytest -s -x --ignore=dataset/ --ignore=test_torch_functional.py .
diff --git a/.github/workflows/verl_unit_test_gpu.yml b/.github/workflows/verl_unit_test_gpu.yml
new file mode 100644
index 00000000000..c304a117169
--- /dev/null
+++ b/.github/workflows/verl_unit_test_gpu.yml
@@ -0,0 +1,47 @@
+name: verl_unit_test_gpu
+
+on:
+  # Trigger the workflow on push or pull request,
+  # but only for the main branch
+  push:
+    branches:
+      - main
+      - v0.*
+  pull_request:
+    branches:
+      - main
+      - v0.*
+    paths:
+      - "**/*.py"
+      - .github/workflows/verl_unit_test_gpu.yml
+      - "!recipe/**/*.py"
+
+# Cancel jobs on the same ref if a new one is triggered
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
+
+# Declare permissions just read content.
+permissions:
+  contents: read
+
+jobs:
+  verl_unit_test_gpu:
+    runs-on: [L20x8]
+    timeout-minutes: 20 # Increase this timeout value as needed
+    container:
+      image: whatcanyousee/verl:ngc-cu124-vllm0.8.3-sglang0.4.5-mcore0.12.0-te2.2
+      options: --gpus all --shm-size=10g
+    steps:
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5.3.0
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install the current repository
+        run: |
+          pip install -e .[test]
+      - name: Running utils tests
+        run: |
+          cd tests/verl/utils
+          pytest -s -x test_torch_functional.py
\ No newline at end of file
diff --git a/tests/verl/utils/test_fs.py b/tests/verl/utils/test_fs.py
new file mode 100644
index 00000000000..7ae85e01aec
--- /dev/null
+++ b/tests/verl/utils/test_fs.py
@@ -0,0 +1,94 @@
+# Copyright 2024 Bytedance Ltd. and/or its affiliates
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+from pathlib import Path
+
+import verl.utils.fs as fs
+
+
+def test_record_and_check_directory_structure(tmp_path):
+    # Create test directory structure
+    test_dir = tmp_path / "test_dir"
+    test_dir.mkdir()
+    (test_dir / "file1.txt").write_text("test")
+    (test_dir / "subdir").mkdir()
+    (test_dir / "subdir" / "file2.txt").write_text("test")
+
+    # Create structure record
+    record_file = fs._record_directory_structure(test_dir)
+
+    # Verify record file exists
+    assert os.path.exists(record_file)
+
+    # Initial check should pass
+    assert fs._check_directory_structure(test_dir, record_file) is True
+
+    # Modify structure and verify check fails
+    (test_dir / "new_file.txt").write_text("test")
+    assert fs._check_directory_structure(test_dir, record_file) is False
+
+
+def test_copy_from_hdfs_with_mocks(tmp_path, monkeypatch):
+    # Mock HDFS dependencies
+    monkeypatch.setattr(fs, "is_non_local", lambda path: True)
+
+    # side_effect will simulate the copy by creating parent dirs + empty file
+    def fake_copy(src: str, dst: str, *args, **kwargs):
+        dst_path = Path(dst)
+        dst_path.parent.mkdir(parents=True, exist_ok=True)
+        dst_path.write_bytes(b"")  # touch an empty file
+
+    monkeypatch.setattr(fs, "copy", fake_copy)  # Mock actual HDFS copy
+
+    # Test parameters
+    test_cache = tmp_path / "cache"
+    hdfs_path = "hdfs://test/path/file.txt"
+
+    # Test initial copy
+    local_path = fs.copy_to_local(hdfs_path, cache_dir=test_cache)
+    expected_path = os.path.join(test_cache, fs.md5_encode(hdfs_path), os.path.basename(hdfs_path))
+    assert local_path == expected_path
+    assert os.path.exists(local_path)
+
+
+def test_always_recopy_flag(tmp_path, monkeypatch):
+    # Mock HDFS dependencies
+    monkeypatch.setattr(fs, "is_non_local", lambda path: True)
+
+    copy_call_count = 0
+
+    def fake_copy(src: str, dst: str, *args, **kwargs):
+        nonlocal copy_call_count
+        copy_call_count += 1
+        dst_path = Path(dst)
+        dst_path.parent.mkdir(parents=True, exist_ok=True)
+        dst_path.write_bytes(b"")
+
+    monkeypatch.setattr(fs, "copy", fake_copy)  # Mock actual HDFS copy
+
+    test_cache = tmp_path / "cache"
+    hdfs_path = "hdfs://test/path/file.txt"
+
+    # Initial copy (always_recopy=False)
+    fs.copy_to_local(hdfs_path, cache_dir=test_cache)
+    assert copy_call_count == 1
+
+    # Force recopy (always_recopy=True)
+    fs.copy_to_local(hdfs_path, cache_dir=test_cache, always_recopy=True)
+    assert copy_call_count == 2
+
+    # Subsequent normal call (always_recopy=False)
+    fs.copy_to_local(hdfs_path, cache_dir=test_cache)
+    assert copy_call_count == 2  # Should not increment
diff --git a/tests/verl/utils/test_torch_functional.py b/tests/verl/utils/test_torch_functional.py
new file mode 100644
index 00000000000..5084e18eac0
--- /dev/null
+++ b/tests/verl/utils/test_torch_functional.py
@@ -0,0 +1,103 @@
+# Copyright 2024 Bytedance Ltd. and/or its affiliates
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+
+import pytest
+import torch
+import torch.distributed as dist
+import torch.multiprocessing as mp
+
+from verl.utils.torch_functional import distributed_masked_mean, distributed_mean_max_min_std
+
+
+def _worker_mean(rank: int, world_size: int, rendezvous_file: str):
+    # 1) set GPU and init NCCL
+    torch.cuda.set_device(rank)
+    dist.init_process_group(
+        backend="nccl",
+        init_method=f"file://{rendezvous_file}",
+        rank=rank,
+        world_size=world_size,
+    )
+
+    # each rank holds tensor [rank+1]
+    local = torch.tensor([float(rank + 1)], device=f"cuda:{rank}")
+    mean, gmax, gmin, gstd = distributed_mean_max_min_std(local, True, True, True)
+
+    values = [float(i + 1) for i in range(world_size)]
+    exp_mean = sum(values) / len(values)
+    exp_max = max(values)
+    exp_min = min(values)
+    var = sum((x - exp_mean) ** 2 for x in values) / (len(values) - 1)
+    exp_std = var**0.5
+
+    # all ranks should see the same result
+    assert torch.allclose(mean.cpu(), torch.tensor(exp_mean)), f"mean@{rank}"
+    assert torch.allclose(gmax.cpu(), torch.tensor(exp_max)), f"max@{rank}"
+    assert torch.allclose(gmin.cpu(), torch.tensor(exp_min)), f"min@{rank}"
+    assert torch.allclose(gstd.cpu(), torch.tensor(exp_std)), f"std@{rank}"
+
+    dist.destroy_process_group()
+
+
+@pytest.mark.parametrize("world_size", [2, 4])
+def test_distributed_mean_max_min_std(world_size, tmp_path):
+    rendezvous_file = str(tmp_path / "rdzv_mean")
+    os.makedirs(os.path.dirname(rendezvous_file), exist_ok=True)
+
+    mp.spawn(
+        fn=_worker_mean,
+        args=(world_size, rendezvous_file),
+        nprocs=world_size,
+        join=True,
+    )
+
+
+def _worker_mask(rank: int, world_size: int, rendezvous_file: str):
+    torch.cuda.set_device(rank)
+    dist.init_process_group(
+        backend="nccl",
+        init_method=f"file://{rendezvous_file}",
+        rank=rank,
+        world_size=world_size,
+    )
+
+    # build per‐rank tensor and mask
+    local_tensor = torch.tensor([rank * 2 + 1.0, rank * 2 + 2.0], device=f"cuda:{rank}")
+    if rank == 0:
+        mask = torch.tensor([1, 0], device=f"cuda:{rank}", dtype=torch.float32)
+    else:
+        mask = torch.tensor([0, 1], device=f"cuda:{rank}", dtype=torch.float32)
+
+    gmean = distributed_masked_mean(local_tensor, mask)
+
+    valid_values = [1.0] + [2 * i + 2.0 for i in range(1, world_size)]
+    expected_mean = sum(valid_values) / len(valid_values)
+    assert torch.allclose(gmean.cpu(), torch.tensor(expected_mean)), f"masked_mean@{rank}"
+
+    dist.destroy_process_group()
+
+
+@pytest.mark.parametrize("world_size", [2, 4])
+def test_distributed_masked_mean(world_size, tmp_path):
+    rendezvous_file = str(tmp_path / "rdzv_mask")
+    os.makedirs(os.path.dirname(rendezvous_file), exist_ok=True)
+
+    mp.spawn(
+        fn=_worker_mask,
+        args=(world_size, rendezvous_file),
+        nprocs=world_size,
+        join=True,
+    )
diff --git a/verl/utils/fs.py b/verl/utils/fs.py
index 52e5c69616e..9d280177cbc 100644
--- a/verl/utils/fs.py
+++ b/verl/utils/fs.py
@@ -18,6 +18,7 @@
 
 import hashlib
 import os
+import shutil
 import tempfile
 
 try:
@@ -39,14 +40,17 @@ def md5_encode(path: str) -> str:
 
 
 def get_local_temp_path(hdfs_path: str, cache_dir: str) -> str:
-    """Return a local temp path that joins cache_dir and basename of hdfs_path
+    """Generate a unique local cache path for an HDFS resource.
+    Creates a MD5-hashed subdirectory in cache_dir to avoid name conflicts,
+    then returns path combining this subdirectory with the HDFS basename.
 
     Args:
-        hdfs_path:
-        cache_dir:
+        hdfs_path (str): Source HDFS path to be cached
+        cache_dir (str): Local directory for storing cached files
 
     Returns:
-
+        str: Absolute local filesystem path in format:
+            {cache_dir}/{md5(hdfs_path)}/{basename(hdfs_path)}
     """
     # make a base64 encoding of hdfs_path to avoid directory conflict
     encoded_hdfs_path = md5_encode(hdfs_path)
@@ -56,21 +60,54 @@ def get_local_temp_path(hdfs_path: str, cache_dir: str) -> str:
     return dst
 
 
-def copy_to_local(src: str, cache_dir=None, filelock=".file.lock", verbose=False) -> str:
-    """Copy src from hdfs to local if src is on hdfs or directly return src.
-    If cache_dir is None, we will use the default cache dir of the system. Note that this may cause conflicts if
-    the src name is the same between calls
+def _record_directory_structure(folder_path):
+    record_file = os.path.join(folder_path, ".directory_record.txt")
+    with open(record_file, "w") as f:
+        for root, dirs, files in os.walk(folder_path):
+            for dir_name in dirs:
+                relative_dir = os.path.relpath(os.path.join(root, dir_name), folder_path)
+                f.write(f"dir:{relative_dir}\n")
+            for file_name in files:
+                if file_name != ".directory_record.txt":
+                    relative_file = os.path.relpath(os.path.join(root, file_name), folder_path)
+                    f.write(f"file:{relative_file}\n")
+    return record_file
+
+
+def _check_directory_structure(folder_path, record_file):
+    if not os.path.exists(record_file):
+        return False
+    existing_entries = set()
+    for root, dirs, files in os.walk(folder_path):
+        for dir_name in dirs:
+            relative_dir = os.path.relpath(os.path.join(root, dir_name), folder_path)
+            existing_entries.add(f"dir:{relative_dir}")
+        for file_name in files:
+            if file_name != ".directory_record.txt":
+                relative_file = os.path.relpath(os.path.join(root, file_name), folder_path)
+                existing_entries.add(f"file:{relative_file}")
+    with open(record_file) as f:
+        recorded_entries = set(f.read().splitlines())
+    return existing_entries == recorded_entries
+
+
+def copy_to_local(src: str, cache_dir=None, filelock=".file.lock", verbose=False, always_recopy=False) -> str:
+    """Copy files/directories from HDFS to local cache with validation.
 
     Args:
-        src (str): a HDFS path of a local path
+        src (str): Source path - HDFS path (hdfs://...) or local filesystem path
+        cache_dir (str, optional): Local directory for cached files. Uses system tempdir if None
+        filelock (str): Base name for file lock. Defaults to ".file.lock"
+        verbose (bool): Enable copy operation logging. Defaults to False
+        always_recopy (bool): Force fresh copy ignoring cache. Defaults to False
 
     Returns:
-        a local path of the copied file
+        str: Local filesystem path to copied resource
     """
-    return copy_local_path_from_hdfs(src, cache_dir, filelock, verbose)
+    return copy_local_path_from_hdfs(src, cache_dir, filelock, verbose, always_recopy)
 
 
-def copy_local_path_from_hdfs(src: str, cache_dir=None, filelock=".file.lock", verbose=False) -> str:
+def copy_local_path_from_hdfs(src: str, cache_dir=None, filelock=".file.lock", verbose=False, always_recopy=False) -> str:
     """Deprecated. Please use copy_to_local instead."""
     from filelock import FileLock
 
@@ -88,10 +125,26 @@ def copy_local_path_from_hdfs(src: str, cache_dir=None, filelock=".file.lock", v
         filelock = md5_encode(src) + ".lock"
         lock_file = os.path.join(cache_dir, filelock)
         with FileLock(lock_file=lock_file):
+            if always_recopy and os.path.exists(local_path):
+                if os.path.isdir(local_path):
+                    shutil.rmtree(local_path, ignore_errors=True)
+                else:
+                    os.remove(local_path)
             if not os.path.exists(local_path):
                 if verbose:
                     print(f"Copy from {src} to {local_path}")
                 copy(src, local_path)
+                if os.path.isdir(local_path):
+                    _record_directory_structure(local_path)
+            elif os.path.isdir(local_path):
+                # always_recopy=False, local path exists, and it is a folder: check whether there is anything missed
+                record_file = os.path.join(local_path, ".directory_record.txt")
+                if not _check_directory_structure(local_path, record_file):
+                    if verbose:
+                        print(f"Recopy from {src} to {local_path} due to missing files or directories.")
+                    shutil.rmtree(local_path, ignore_errors=True)
+                    copy(src, local_path)
+                    _record_directory_structure(local_path)
         return local_path
     else:
         return src
diff --git a/verl/utils/torch_functional.py b/verl/utils/torch_functional.py
index 2d42b25d97a..344ec65eb6d 100644
--- a/verl/utils/torch_functional.py
+++ b/verl/utils/torch_functional.py
@@ -26,6 +26,7 @@
 from torch import nn
 from torch.optim import Optimizer
 from torch.optim.lr_scheduler import LambdaLR
+from transformers import PreTrainedTokenizer
 
 try:
     from flash_attn.ops.triton.cross_entropy import cross_entropy_loss
@@ -247,10 +248,10 @@ def pad_sequence_to_length(tensors, max_seq_len, pad_token_id, left_pad=False):
     pad a 2D tensors (e.g. responses, logprobs) in the last dim to max_seq_length.
     input shape: [bs, seq_length]
     output shape: [bs, max_seq_length]
-    (0, max_seq_len - tensors.shape[-1]) means right pad to max_seq_length and no left pad
     """
     if tensors.shape[-1] >= max_seq_len:
         return tensors
+    # (0, max_seq_len - tensors.shape[-1]) means right pad to max_seq_length and no left pad
     pad_tuple = (max_seq_len - tensors.shape[-1], 0) if left_pad else (0, max_seq_len - tensors.shape[-1])
     return F.pad(tensors, pad_tuple, "constant", pad_token_id)
 
@@ -263,8 +264,18 @@ def postprocess_data(
     left_pad=True,
     truncation="error",
 ):
-    """
-    input_data is the output from tokenizer.
+    """Process tokenizer outputs to consistent shapes via padding/truncation.
+
+    Args:
+        input_ids: Token indices [batch_size, seq_len]
+        attention_mask: Mask [batch_size, seq_len]
+        max_length: Target sequence length
+        pad_token_id: Padding token ID
+        left_pad: Pad left if True
+        truncation: "left", "right" or "error"
+
+    Returns:
+        (input_ids, attention_mask) padded/truncated to max_length
     """
     assert truncation in ["left", "right", "error"]
     assert input_ids.ndim == 2
@@ -289,6 +300,27 @@ def postprocess_data(
     return input_ids, attention_mask
 
 
+def tokenize_and_postprocess_data(prompt: str, tokenizer: PreTrainedTokenizer, max_length: int, pad_token_id: int, left_pad=True, truncation="error"):
+    """Tokenize text and process outputs to consistent tensor shapes.
+
+    Args:
+        prompt: Input text to tokenize
+        tokenizer: HuggingFace tokenizer instance
+        max_length: Target sequence length
+        pad_token_id: Padding token ID
+        left_pad: Pad left if True
+        truncation: Truncation strategy ("left"/"right"/"error")
+
+    Returns:
+        Tuple of (input_ids, attention_mask) from postprocess_data
+    """
+    input_data = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
+    input_ids = input_data["input_ids"]
+    attention_mask = input_data["attention_mask"]
+
+    return postprocess_data(input_ids, attention_mask, max_length, pad_token_id, left_pad, truncation)
+
+
 def remove_pad_token(input_ids: torch.Tensor, attention_mask: torch.Tensor):
     """Remove the pad token.
 
@@ -566,3 +598,68 @@ def check_cuda_is_available():
         raise RuntimeError("CUDA must be initialized before importing this module.")
 
     yield
+
+
+def distributed_mean_max_min_std(local_tensor, compute_max=True, compute_min=True, compute_std=True):
+    """Compute distributed statistics across all processes.
+
+    Args:
+        local_tensor: Tensor containing local values
+        compute_max: Include maximum value calculation
+        compute_min: Include minimum value calculation
+        compute_std: Include standard deviation calculation
+
+    Returns:
+        Tuple containing (mean, max, min, std) in this order. None for disabled metrics.
+    """
+    # Sum the local tensor across all processes
+    local_sum = torch.sum(local_tensor)
+    local_num = torch.tensor(torch.numel(local_tensor), device="cuda")
+
+    torch.distributed.all_reduce(local_sum, op=torch.distributed.ReduceOp.SUM)
+    torch.distributed.all_reduce(local_num, op=torch.distributed.ReduceOp.SUM)
+
+    global_mean = local_sum / local_num
+
+    if compute_max:
+        local_max = torch.max(local_tensor)
+        torch.distributed.all_reduce(local_max, op=torch.distributed.ReduceOp.MAX)
+    else:
+        local_max = None
+
+    if compute_min:
+        local_min = torch.min(local_tensor)
+        torch.distributed.all_reduce(local_min, op=torch.distributed.ReduceOp.MIN)
+    else:
+        local_min = None
+
+    if compute_std:
+        square_diff = torch.sum(torch.pow(local_tensor - global_mean, 2))
+        torch.distributed.all_reduce(square_diff, op=torch.distributed.ReduceOp.SUM)
+        global_std = torch.sqrt(square_diff / (local_num - 1))
+    else:
+        global_std = None
+
+    return global_mean, local_max, local_min, global_std
+
+
+def distributed_masked_mean(local_tensor, local_mask):
+    """Compute global mean of non-masked elements across distributed processes.
+
+    Args:
+        local_tensor (torch.Tensor): Input tensor with local values
+        local_mask (torch.Tensor): Binary mask (1=valid, 0=ignore) matching local_tensor shape
+
+    Returns:
+        torch.Tensor: Global mean of all valid elements across processes
+    """
+    local_tensor = local_tensor * local_mask
+
+    local_sum = torch.sum(local_tensor)
+    local_num = torch.sum(local_mask)
+
+    torch.distributed.all_reduce(local_sum, op=torch.distributed.ReduceOp.SUM)
+    torch.distributed.all_reduce(local_num, op=torch.distributed.ReduceOp.SUM)
+
+    global_mean = local_sum / local_num
+    return global_mean


Reviews:
user: login: eric-haibin-lin
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-05-08T23:30:01Z
user: login: hongpeng-guo
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-08T23:32:49Z
user: login: eric-haibin-lin
body: last few comments on docs. we can merge it once docs are done
state: DISMISSED
author_association: COLLABORATOR
submitted_at: 2025-05-09T16:43:26Z
user: login: eric-haibin-lin
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-09T23:50:43Z

Review comments:
diff_hunk: @@ -27,11 +27,11 @@ permissions:
 
 jobs:
   verl_unit_test:
-    runs-on: ubuntu-latest
-    timeout-minutes: 10 # Increase this timeout value as needed
-    strategy:
-      matrix:
-        python-version: ["3.10"]
+    runs-on: [L20x8]
+    timeout-minutes: 20 # Increase this timeout value as needed
+    container:
+      image: whatcanyousee/verl:ngc-cu124-vllm0.8.3-sglang0.4.5-mcore0.12.0-te2.2
path: .github/workflows/verl_unit_test.yml
user: login: eric-haibin-lin
body: hmmm these tests used to be cpu only
created_at: 2025-05-08T23:29:57Z
author_association: COLLABORATOR
diff_hunk: @@ -27,11 +27,11 @@ permissions:
 
 jobs:
   verl_unit_test:
-    runs-on: ubuntu-latest
-    timeout-minutes: 10 # Increase this timeout value as needed
-    strategy:
-      matrix:
-        python-version: ["3.10"]
+    runs-on: [L20x8]
+    timeout-minutes: 20 # Increase this timeout value as needed
+    container:
+      image: whatcanyousee/verl:ngc-cu124-vllm0.8.3-sglang0.4.5-mcore0.12.0-te2.2
path: .github/workflows/verl_unit_test.yml
user: login: hongpeng-guo
body:  yeah, as I added the torch_function_tests which has a hard requirement of `cuda` and torch.dist. Maybe I can start a new pipeline `verl_unit_test_gpu`
created_at: 2025-05-08T23:32:49Z
author_association: COLLABORATOR
diff_hunk: @@ -56,7 +57,38 @@ def get_local_temp_path(hdfs_path: str, cache_dir: str) -> str:
     return dst
 
 
-def copy_to_local(src: str, cache_dir=None, filelock=".file.lock", verbose=False) -> str:
+def _record_directory_structure(folder_path):
+    record_file = os.path.join(folder_path, ".directory_record.txt")
+    with open(record_file, "w") as f:
+        for root, dirs, files in os.walk(folder_path):
+            for dir_name in dirs:
+                relative_dir = os.path.relpath(os.path.join(root, dir_name), folder_path)
+                f.write(f"dir:{relative_dir}\n")
+            for file_name in files:
+                if file_name != ".directory_record.txt":
+                    relative_file = os.path.relpath(os.path.join(root, file_name), folder_path)
+                    f.write(f"file:{relative_file}\n")
+    return record_file
+
+
+def _check_directory_structure(folder_path, record_file):
+    if not os.path.exists(record_file):
+        return False
+    existing_entries = set()
+    for root, dirs, files in os.walk(folder_path):
+        for dir_name in dirs:
+            relative_dir = os.path.relpath(os.path.join(root, dir_name), folder_path)
+            existing_entries.add(f"dir:{relative_dir}")
+        for file_name in files:
+            if file_name != ".directory_record.txt":
+                relative_file = os.path.relpath(os.path.join(root, file_name), folder_path)
+                existing_entries.add(f"file:{relative_file}")
+    with open(record_file) as f:
+        recorded_entries = set(f.read().splitlines())
+    return existing_entries == recorded_entries
+
+
+def copy_to_local(src: str, cache_dir=None, filelock=".file.lock", verbose=False, always_recopy=False) -> str:
     """Copy src from hdfs to local if src is on hdfs or directly return src.
     If cache_dir is None, we will use the default cache dir of the system. Note that this may cause conflicts if
     the src name is the same between calls
path: verl/utils/fs.py
user: login: eric-haibin-lin
body: please add the doc of these arguments (and the prev missing ones). thanks! 
created_at: 2025-05-09T16:41:19Z
author_association: COLLABORATOR
diff_hunk: @@ -289,6 +290,17 @@ def postprocess_data(
     return input_ids, attention_mask
 
 
+def tokenize_and_postprocess_data(prompt: str, tokenizer: PreTrainedTokenizer, max_length: int, pad_token_id: int, left_pad=True, truncation="error"):
+    """
+    input_data is the output from tokenizer.
path: verl/utils/torch_functional.py
user: login: eric-haibin-lin
body: please briefly describe what post processing is done, and same for the postprocess_data api
created_at: 2025-05-09T16:42:33Z
author_association: COLLABORATOR
diff_hunk: @@ -566,3 +578,57 @@ def check_cuda_is_available():
         raise RuntimeError("CUDA must be initialized before importing this module.")
 
     yield
+
+
+def distributed_mean_max_min_std(local_tensor, compute_max=True, compute_min=True, compute_std=True):
+    """
+    Compute the mean of a tensor across all processes in a distributed setup.
+
+    Args:
+        local_tensor (torch.Tensor): The local tensor on each process.
+
+    Returns:
+        torch.Tensor: The mean of the tensor across all processes.
+    """
+    # Sum the local tensor across all processes
+    local_sum = torch.sum(local_tensor)
+    local_num = torch.tensor(torch.numel(local_tensor), device="cuda")
+
+    torch.distributed.all_reduce(local_sum, op=torch.distributed.ReduceOp.SUM)
+    torch.distributed.all_reduce(local_num, op=torch.distributed.ReduceOp.SUM)
+
+    global_mean = local_sum / local_num
+
+    if compute_max:
+        local_max = torch.max(local_tensor)
+        torch.distributed.all_reduce(local_max, op=torch.distributed.ReduceOp.MAX)
+    else:
+        local_max = None
+
+    if compute_min:
+        local_min = torch.min(local_tensor)
+        torch.distributed.all_reduce(local_min, op=torch.distributed.ReduceOp.MIN)
+    else:
+        local_min = None
+
+    if compute_std:
+        square_diff = torch.sum(torch.pow(local_tensor - global_mean, 2))
+        torch.distributed.all_reduce(square_diff, op=torch.distributed.ReduceOp.SUM)
+        global_std = torch.sqrt(square_diff / (local_num - 1))
+    else:
+        global_std = None
+
+    return global_mean, local_max, local_min, global_std
+
+
+def distributed_masked_mean(local_tensor, local_mask):
path: verl/utils/torch_functional.py
user: login: eric-haibin-lin
body: if it's an API, pls add some docs
created_at: 2025-05-09T16:43:07Z
author_association: COLLABORATOR

PR comments:
user: login: hongpeng-guo
created_at: 2025-05-09T05:32:50Z
body: @eric-haibin-lin updated, can be reviewed again.
author_association: COLLABORATOR
user: login: hongpeng-guo
created_at: 2025-05-09T20:33:38Z
body: > last few comments on docs. we can merge it once docs are done

updated.
author_association: COLLABORATOR