[tab] README_arrayrecord.md (markdown)
# ArrayRecord Dataset Generation

This directory contains scripts to generate ArrayRecord datasets compatible with the `train_tokenizer.py` script.

## Overview

The `generate_arrayrecord_dataset.py` script creates synthetic video data in ArrayRecord format that can be used for training the VQ-VAE tokenizer. ArrayRecord is a high-performance data format that provides efficient random access and is well-suited for machine learning training.

## Features

- **Synthetic Video Generation**: Creates moving patterns with temporal consistency
- **Configurable Dimensions**: Supports custom video dimensions (seq_len, height, width, channels)
- **Efficient Storage**: Uses ArrayRecord format for fast random access
- **Verification**: Includes dataset verification functionality
- **Compatibility**: Designed to work with the existing `train_tokenizer.py` script

## Installation

Install the required dependencies:

```bash
pip install -r requirements.txt
```

## Usage

### Basic Usage

Generate a default dataset:

```bash
python generate_arrayrecord_dataset.py
```

This creates:
- 1000 videos with 16 frames each
- Frame dimensions: 90×160×3
- Output directory: `data_arrayrecord/coinrun`
- 100 videos per ArrayRecord file

### Custom Configuration

Generate a custom dataset:

```bash
python generate_arrayrecord_dataset.py \
    --output_dir data_arrayrecord/custom \
    --num_videos 5000 \
    --seq_len 32 \
    --height 128 \
    --width 256 \
    --channels 3 \
    --records_per_file 200 \
    --seed 123 \
    --verify
```

### Parameters

- `--output_dir`: Output directory for ArrayRecord files (default: `data_arrayrecord/coinrun`)
- `--num_videos`: Total number of videos to generate (default: 1000)
- `--seq_len`: Number of frames per video (default: 16)
- `--height`: Frame height (default: 90)
- `--width`: Frame width (default: 160)
- `--channels`: Number of color channels (default: 3)
- `--records_per_file`: Number of records per ArrayRecord file (default: 100)
- `--seed`: Random seed for reproducibility (default: 42)
- `--verify`: Verify the generated dataset (default: False)

## Dataset Format

Each ArrayRecord file contains serialized Python dictionaries with the following structure:

```python
{
    "videos": np.ndarray  # Shape: (seq_len, height, width, channels), dtype: float32
}
```

### Video Data Specifications

- **Data Type**: `float32`
- **Value Range**: [0, 1] (normalized pixel values)
- **Shape**: `(seq_len, height, width, channels)`
- **Temporal Consistency**: Moving patterns with smooth transitions

## Integration with Training Script

The generated ArrayRecord files are designed to be compatible with `train_tokenizer.py`. However, you may need to modify the dataloader to handle ArrayRecord format instead of TFRecord format.

### Example Training Command

```bash
python train_tokenizer.py \
    --data_dir data_arrayrecord/coinrun \
    --num_steps 1000 \
    --batch_size 8 \
    --log false \
    --ckpt_dir checkpoints
```

## Verification

The script includes a verification function that checks:

- File integrity
- Data shapes and types
- Value ranges
- Sample loading

Run verification with:

```bash
python generate_arrayrecord_dataset.py --verify
```

## Example Usage Script

Run the complete example:

```bash
python example_usage.py
```

This script demonstrates:
1. Dataset generation
2. Verification
3. Training command examples

## Performance Considerations

- **Memory Usage**: ArrayRecord files are memory-efficient and support streaming
- **I/O Performance**: Optimized for random access and parallel reading
- **Scalability**: Files can be easily sharded across multiple workers

## Troubleshooting

### Common Issues

1. **Import Error**: Ensure `array-record` is installed correctly
2. **Memory Issues**: Reduce `records_per_file` for large datasets
3. **Disk Space**: Monitor output directory size for large datasets

### Verification Output

Successful verification should show:
```
Verifying dataset in data_arrayrecord/coinrun...
Found 10 ArrayRecord files

Verifying file: videos_0000.array_record
  Record 0: video shape = (16, 90, 160, 3), dtype = float32
    Value range: [0.100, 1.000]
    Mean: 0.234
```

## File Structure

```
data_arrayrecord/
└── coinrun/
    ├── videos_0000.array_record
    ├── videos_0001.array_record
    ├── ...
    └── videos_0009.array_record
```

Each file contains multiple video records, and the training script can efficiently load from multiple files in parallel.