LiberalMind Video VAE
Learning compressed representations of the visual world.
LiberalMind Video VAE is a video representation learning model developed as part of the LiberalMind embodied intelligence research initiative.
The model is trained to encode and reconstruct short video sequences with a Variational Autoencoder (VAE). Its training objective combines pixel-level reconstruction, perceptual similarity, and latent regularization.
The checkpoint was produced using PyTorch on Google Cloud TPU with the PyTorch/XLA stack and SPMD data parallelism.
This repository is primarily a model/checkpoint repository. It documents the training setup and launch procedure, but it does not provide a self-contained copy of the complete training implementation. The training and model-definition scripts (train.py and vae.py) must be obtained or supplied separately.
Model Overview
| Property | Configuration |
|---|---|
| Organization | LiberalMind |
| Model type | Video Variational Autoencoder |
| Framework | PyTorch |
| Training accelerator | Google Cloud TPU |
| TPU backend | PyTorch/XLA |
| Parallelization | SPMD data parallelism |
| Input resolution | 256 Γ 256 |
| Temporal sequence length | 17 frames |
| Input channels | RGB (3) |
| Training precision | BF16 |
| Optimizer | AdamW |
| Learning rate | 1e-4 |
| Default batch size | 8 |
| Default training target | 200,000 steps |
| Perceptual loss | LPIPS (VGG) |
What This Model Is For
The model learns a compact visual representation by reconstructing short video clips. The current implementation is focused on video reconstruction and representation learning.
Potential research applications include:
- Video representation learning.
- Temporal visual understanding.
- Embodied AI and robot perception.
- Components for autonomous agents.
- Research toward future world models.
Important distinction: this checkpoint is a video reconstruction model. It does not, by itself, provide action-conditioned future-state prediction or a complete predictive world model.
Repository Contents and Required Source Files
The model repository is intended to distribute the trained checkpoint and related artifacts. Depending on the uploaded revision, it may contain files such as:
vae-model/
βββ README.md
βββ last.pt # trained checkpoint
βββ log.txt # training log, if included
βββ rec_*.jpg # reconstruction previews, if included
The following Python source files are required to reproduce training or instantiate the model, but are not included as part of the checkpoint-only repository setup described here:
vae.py # VAE architecture
train.py # TPU training script
The training script imports the model with:
from vae import VAE
Therefore, train.py and vae.py must be present in the same directory, or that directory must be added to PYTHONPATH. Downloading the checkpoint alone is not enough to run training or load the model architecture.
Training Setup
Hardware and Environment
The documented training pipeline targets:
- A Linux Google Cloud TPU VM.
- Python 3.10 or a compatible version.
- A working Google Cloud TPU runtime.
- PyTorch and PyTorch/XLA with compatible versions.
- Sufficient host RAM for video decoding and data loading.
- A local video dataset on the TPU VM.
One compatible installation set for the documented environment is:
python3.10 -m venv ~/vae_env
source ~/vae_env/bin/activate
pip install --upgrade pip
pip install torch==2.8.0 "torch_xla[tpu]==2.8.0"
pip install torchvision==0.23.0
pip install numpy av lpips huggingface_hub
Check that the TPU runtime is available before starting:
import torch
import torch_xla
import torch_xla.runtime as xr
print("PyTorch:", torch.__version__)
print("PyTorch/XLA:", torch_xla.__version__)
xr.use_spmd()
print("TPU devices:", xr.global_runtime_device_count())
The TPU runtime must initialize successfully. The global batch size should be divisible by the number of TPU devices used by the SPMD program.
Dataset Layout
The documented training pipeline expects MP4 files in:
~/ego256/
Example layout:
/home/user/
βββ ego256/
β βββ video_0001.mp4
β βββ video_0002.mp4
β βββ ...
βββ vae.py # supplied separately
βββ train.py # supplied separately
βββ vae_run/ # training output directory
Dataset requirements:
- MP4 video files with RGB-compatible video streams.
- Resolution of exactly 256 Γ 256 pixels.
- At least 20 decodable frames per video.
- Valid frame-rate metadata.
The documented loader does not resize input videos. Preprocess videos to 256 Γ 256 before training. Each training sample uses 17 consecutive frames. Video sampling is weighted by the number of frames, and random horizontal flipping is applied with probability 0.5.
Launching Training
This section describes how to launch the training script when the required source files are already available on the TPU VM. The launch command does not download or create train.py or vae.py.
1. Check the source files
Make sure both files exist:
ls -lh ~/train.py ~/vae.py
If they are stored in a project directory instead of your home directory, use their actual paths and make sure Python can import vae.py from train.py.
2. Activate the environment
source ~/vae_env/bin/activate
3. Start training
python ~/train.py
The documented default configuration is:
T = 17
RES = 256
BATCH = 8
LR = 1e-4
WARM = 1000
STEPS = 200000
W_LPIPS = 0.5
W_KL = 1e-6
The script supports the following environment-variable overrides:
BATCH=8 STEPS=200000 python ~/train.py
A short test run:
BATCH=4 STEPS=100 python ~/train.py
A longer run:
BATCH=8 STEPS=300000 python ~/train.py
STEPS is the target total training step, not the number of extra steps to add to an existing checkpoint. Keep BATCH divisible by the number of TPU devices used.
4. Training output and resume
The documented output directory is:
~/vae_run/
Expected artifacts include:
vae_run/
βββ last.pt
βββ log.txt
βββ rec_*.jpg
The checkpoint is periodically saved (documented interval: every 2,000 steps). The checkpoint format is:
{
"model": model.state_dict(),
"opt": opt.state_dict(),
"sched": sched.state_dict(),
"step": step,
}
The original training pipeline is documented to resume automatically from ~/vae_run/last.pt, restoring the model, optimizer, scheduler, and step. Restarting the matching training script continues from the saved checkpoint:
STEPS=200000 python ~/train.py
To start a new run from scratch, move the existing checkpoint out of ~/vae_run/last.pt before launching. Resume behavior depends on the separately supplied train.py; it cannot be guaranteed by this model repository alone.
5. Monitoring
Training logs are written to:
~/vae_run/log.txt
The documented metrics are:
| Metric | Meaning |
|---|---|
loss |
Combined training objective |
l1 |
Pixel reconstruction error |
lpips |
Perceptual difference |
kl |
Latent regularization |
s/it |
Average seconds per iteration |
Reconstruction previews are documented to be saved every 1,000 steps. They show original and reconstructed frames and can be used for qualitative monitoring.
Using the Checkpoint
Download from Hugging Face
The checkpoint repository is:
LiberalMind/vae-model
For a private repository, authenticate first:
hf auth login
Download last.pt:
from huggingface_hub import hf_hub_download
checkpoint_path = hf_hub_download(
repo_id="LiberalMind/vae-model",
filename="last.pt",
token=True,
)
print("Checkpoint:", checkpoint_path)
You also need the matching vae.py file to instantiate the architecture. The checkpoint does not contain the Python model class definition.
Load the Model
import torch
from vae import VAE
checkpoint = torch.load(
checkpoint_path,
map_location="cpu",
weights_only=False,
)
model = VAE()
model.load_state_dict(checkpoint["model"])
model.eval()
print("Model loaded successfully")
print("Training step:", checkpoint["step"])
The example above loads the model on CPU. For GPU or TPU inference, move the model and input tensors to the corresponding device and use a compatible runtime.
Input Shape
The documented forward interface is:
rec, mu, logvar = model(x)
Expected input tensor shape:
[B, 3, 17, 256, 256]
where B is the batch size, 3 is the RGB channel count, and 17 is the number of video frames.
The exact latent dimensionality and encoder/decoder architecture are defined in the separately supplied vae.py source file.
Current Scope and Limitations
The documented implementation provides:
- Video-based VAE training.
- Temporal input sequences.
- Pixel and perceptual reconstruction objectives.
- KL-regularized latent representations.
- TPU SPMD execution and BF16 training.
- Periodic checkpointing and reconstruction previews.
The documented pipeline does not include:
- A separate validation dataset.
- A quantitative validation benchmark.
- Future-frame prediction.
- Action-conditioned dynamics.
- A real-time video generation interface.
last.pt is the latest periodic checkpoint, not a validation-selected best checkpoint.
About LiberalMind
LiberalMind is an independent AI research initiative exploring intelligent models, autonomous systems, and embodied intelligence.
Our research focuses on models that can perceive, learn, represent, and interact with the physical world.
Organization: LiberalMind
Model repository: LiberalMind/vae-model
Framework: PyTorch
Training hardware: Google Cloud TPU
Research direction: Video Representation Learning Β· Embodied AI Β· World Modeling
Building intelligence beyond the screen.