Multimodal Yield Estimation

A transformer-based model that estimates plot-level corn yield (bu/acre) from weekly text captions of field conditions (weather, soil chemistry, topography and treatment) and UAS RGB images of the plot, at any point in the growing season. This repository contains the trained checkpoint, inference scripts that compute the Qwen3 text and DINOv3 image embeddings and predict yield, and the training code.

Tags: Foundation-AI

License

License

References

  • Qwen/Qwen3-Embedding-0.6B: text embedding model used for the weekly captions (1024-d).
  • facebook/dinov3-vitb16-pretrain-lvd1689m: image backbone used for the UAS plot tiles (768-d).
  • Daymet: source of the daily weather variables summarized in the captions.
  • Hugging Face custom models: how trust_remote_code=True loading works.
  • Cutoff: the number of weeks after planting the model is allowed to see. Week 12 means the prediction uses captions and images from the first 12 weeks only.
  • UAS: Uncrewed Aircraft System (drone) imagery.

Acknowledgements

National Science Foundation (NSF) funded AI institute for Intelligent Cyberinfrastructure with Computational Learning in the Environment (ICICLE) (OAC 2112606)

Issue reporting

For questions, issues, or support, please contact:

Sarikaa Sridhar sridhar.86@osu.edu


Tutorials

Overview

Task Plot-level corn yield regression (bu/acre)
Text input One caption per week after planting, embedded with Qwen/Qwen3-Embedding-0.6B (1024-d)
Image input UAS RGB plot tiles (.tif) with capture dates, embedded with facebook/dinov3-vitb16-pretrain-lvd1689m (768-d)
Season length 22 weeks (150 days) after planting
Cutoffs Trained and evaluated at weeks 4, 8, 12, 16, 20, 22; any week 1–22 can be estimated

Prerequisites

  • Python 3.11 or newer
  • A GPU is recommended. On CPU, the inference scripts take several minutes because they run Qwen3 and DINOv3.

Getting Started: your first prediction

  1. Clone the repository and install the dependencies:

    git clone https://huggingface.co/ICICLE-AI/multimodal-yield-estimation
    cd multimodal-yield-estimation
    pip install -r requirements.txt
    
  2. Point the example at your images. examples/sample_input.json contains real weekly captions for one plot, but the UAS .tif tiles are not included. Each image entry has a placeholder path such as /path/to/images/northwest_bftb_om_corn_rgb_101_20240530.tif; replace /path/to/images/ with the folder holding your tiles (or remove the images list to predict from text only).

  3. Run single-sample inference at week 12:

    python scripts/inference_single_hf.py \
      --hf_model_dir ICICLE-AI/multimodal-yield-estimation \
      --sample_json examples/sample_input.json \
      --week 12 \
      --output_json prediction.json
    
  4. Read the result, printed and saved to prediction.json:

    {
      "sample_id": "northwest_bftb_2024_plot101",
      "week": 12,
      "has_image_by_week": true,
      "image_age_weeks": 0,
      "yield_pred_bu_per_acre": 175.66
    }
    

How-To Guides

Installation

pip install -r requirements.txt

Tested versions:

torch==2.12.1
torchvision==0.27.1
transformers==5.13.0
sentence-transformers==5.7.0
safetensors==0.8.0
numpy==2.4.4
pandas==3.0.3
scikit-learn==1.9.0
pillow==12.2.0

Prepare Input Data

Input format

Both inference scripts use the same per-plot sample format:

{
  "sample_id": "plot_001",
  "planting_date": "2024-05-16",
  "week": 12,
  "texts": [
    "<caption for week 0>",
    "<caption for week 1>",
    "..."
  ],
  "images": [
    {"path": "/path/to/images/plot_001_20240530.tif", "date": "2024-05-30"},
    {"path": "/path/to/images/plot_001_20240606.tif", "date": "2024-06-06"}
  ]
}
Field Required Description
planting_date yes YYYY-MM-DD. Weeks are counted from this date.
week yes, unless --week is passed Cutoff week, 1–22.
texts yes Weekly captions starting at week 0 (planting week). At least week captions are needed; at most 22 are used.
images no List of {path, date}. Order does not matter. An empty or missing list means no images.
sample_id no Copied to the output.
  • Single inference: one JSON object (see examples/sample_input.json).
  • Batch inference: a JSONL file with one object per line (see examples/sample_batch.jsonl).

Write the weekly captions

Week w covers days 7w to 7w+6 after planting; the last week (week 21) covers days 147–149. Each caption describes:

  • 11 weekly weather variables (Daymet): day length, precipitation total, solar radiation, maximum, minimum and mean temperature, vapor pressure, GDD accumulation, and 3-, 7- and 14-day precipitation averages. Values are weekly means, except precipitation and GDD, which are weekly sums.
  • 23 soil variables: pH, buffer pH, organic matter, CEC, P, K, Mg, Ca, K/Ca/Mg TE, S, Zn, Fe, Mn, Cu, B, Na, clay, sand, silt, bulk density and available water capacity.
  • Topography: elevation and slope.
  • Pre-plant soil moisture proxy: mean precipitation over the 14 days before planting.
  • Hybrid/treatment identifier.

Each numeric value is converted to low, medium or high using tertile thresholds computed on the full corn caption dataset (not available when missing). The hybrid/treatment value is written as is.

Example caption (one week of one plot):

Weekly mean day length: medium. Weekly precipitation total: low. Weekly mean solar radiation: high. Weekly mean maximum temperature: high. Weekly mean minimum temperature: medium. Weekly mean vapor pressure: medium. Weekly mean temperature: high. Weekly GDD accumulation: high. Weekly mean 3-day precipitation average: medium. Weekly mean 7-day precipitation average: medium. Weekly mean 14-day precipitation average: high. Soil pH: low. Buffer pH: not available. Soil organic matter: low. Soil cation exchange capacity: high. Soil phosphorus: medium. Soil potassium: high. Soil magnesium: high. Soil calcium: high. Soil potassium TE: low. Soil calcium TE: high. Soil magnesium TE: low. Soil sulfur: high. Soil zinc: medium. Soil iron: high. Soil manganese: low. Soil copper: high. Soil boron: medium. Soil sodium: medium. Soil clay: high. Soil sand: medium. Soil silt: low. Soil bulk density: medium. Soil available water capacity: low. Elevation: low. Slope: low. Pre-plant soil moisture proxy: high. Hybrid/treatment: 100.0.

The model only learned this template. Captions should use the same wording, order and categories; free-form text produces embeddings the model has not seen. The caption generator and the tertile thresholds are not included in this repository.

Prepare the UAS plot tiles

  • RGB plot tiles (.tif) cropped from UAS orthomosaics, one tile per plot per flight date.
  • Any size works: the DINOv3 image processor resizes and normalizes each image.
  • Each image needs its capture date.

Run Inference

Both scripts take raw captions and image files, compute the Qwen3 and DINOv3 embeddings, place the images on the weekly grid and predict yield. --hf_model_dir accepts a local checkpoint folder (use . from the repository root) or a Hub repo ID. The model is trained on weather/soil text caption and UAS plot images with 29.8 bu/acre RMSE yield error at evaluation.

Single sample

python scripts/inference_single_hf.py \
  --hf_model_dir ICICLE-AI/multimodal-yield-estimation \
  --sample_json examples/sample_input.json \
  --week 12 \
  --output_json prediction.json

Batch

python scripts/inference_batch_hf.py \
  --hf_model_dir ICICLE-AI/multimodal-yield-estimation \
  --input_jsonl examples/sample_batch.jsonl \
  --output_json predictions.json \
  --output_csv predictions.csv
  • The JSON output is a list with one object per sample, using the same fields as the single-sample output. --output_csv is optional.
  • Each sample uses its own week; pass --week to use the same week for all samples.
Option Description
--hf_model_dir Local checkpoint folder or Hub repo ID
--sample_json / --input_jsonl Input file
--week Cutoff week (1–22); overrides the sample's week
--output_json Output JSON path (output folder must exist)
--output_csv Batch only: optional CSV output
--batch_size Batch only: model batch size (default 64)

On a SLURM cluster

multimodal_yield_estimation.slurm runs both inference scripts on one GPU. Fill in --account, the conda paths and the repository path before submitting:

sbatch multimodal_yield_estimation.slurm

Load the Model from Python

The model uses custom code, so load it with trust_remote_code=True. The model takes embeddings, not raw text or images; use data/embeddings.py (as the inference scripts do) to build them. Run this from the repository root so data/ can be imported.

import torch
from transformers import AutoModel

from data.embeddings import ImageEmbedder, TextEmbedder, embed_samples

model = AutoModel.from_pretrained(
    "ICICLE-AI/multimodal-yield-estimation", trust_remote_code=True
).eval()

x = embed_samples(
    [sample],  # dict in the input format above
    TextEmbedder(model.config.text_model),
    ImageEmbedder(model.config.image_model),
)

with torch.no_grad():
    out = model(
        text=torch.from_numpy(x["text"]),
        image=torch.from_numpy(x["image"]),
        img_mask=torch.from_numpy(x["img_mask"]),
        img_age=torch.from_numpy(x["img_age"]),
        cutoff=int(x["week"][0]),
    )

print(out.predictions)  # bu/acre

Model inputs:

Input Shape Description
text [B, T, 1024] Raw Qwen3 caption embeddings
image [B, T, 768] Raw DINOv3 features (any value where img_mask is 0)
img_mask [B, T] 1 if an image is available at or before that week
img_age [B, T] Weeks since that image was taken
cutoff int Number of weeks to use (defaults to T)

Normalization statistics are stored in config.json, so inputs are passed un-normalized. The output has predictions (bu/acre) and logits (standardized yield).

Train on Your Own Data

  1. Build an .npz file of embeddings with these arrays:

    Array Shape Description
    text [N, 22, 1024] float32 Weekly Qwen3 caption embeddings
    image [N, 22, 768] float32 DINOv3 feature of the latest image at or before each week (zeros where none)
    img_mask [N, 22] float32 1 if an image is available at or before that week
    img_age [N, 22] float32 Weeks since that image was taken
    yield_ [N] float32 Yield (bu/acre)
    farm_field, plot, planting_date, row_key [N] Plot metadata

    The image arrays can be built with images_to_weeks() in data/multimodal_dataset.py, which is the same function the inference scripts use.

  2. Train. Your data needs at least two farm_field values; pass --test_field <name> to choose which one is held out.

    python scripts/train_multimodal_hf.py \
      --data /path/to/embeddings.npz \
      --out_dir checkpoints
    

    Other options: --cutoffs, --epochs, --patience, --lr, --batch_size, --d_model, --num_layers, --dropout, --input_dropout, --seed.

  3. To train on SLURM, replace the inference commands in multimodal_yield_estimation.slurm with:

    python scripts/train_multimodal_hf.py --data /path/to/your.npz --out_dir checkpoints
    
  4. Use the new checkpoint by passing --hf_model_dir checkpoints to the inference scripts.


Explanation

Problem Description

Given what is known about a corn plot up to a certain week of the season, estimate its final yield.

What is known each week is described in two ways:

  • Text: a caption summarizing that week's weather and the plot's static soil, topography and treatment information.
  • Image: UAS RGB images of the plot, taken on flight dates that do not line up with weeks (every 1–2 weeks during the season).

The model gives one yield estimate per plot for the requested cutoff week.

Placing Images on the Weekly Grid

The inference scripts do this automatically:

  • Image week = (date − planting_date).days // 7. Images before planting or after week 21 are ignored.
  • Each week uses the latest image taken at or before that week (forward-fill). If two images fall in the same week, the later one is used.
  • Weeks before the first image have no image (has_image = 0).
  • The scripts also compute the image age: the number of weeks since the image in use was taken (0 = new image that week).

Example (flights in weeks 2, 3, 4, 6, 7, 9, 11, 14 and 15):

week:       0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
has_image:  0 0 1 1 1 1 1 1 1 1  1  1  1  1  1  1  1  1  1  1  1  1
image_age:  0 0 0 0 0 1 0 0 1 0  1  0  1  2  0  0  1  2  3  4  5  6

Multi-Temporal Yield Estimation

  • Each week after planting is represented by a Qwen3 embedding of that week's caption and a DINOv3 embedding of the most recent UAS image, concatenated with a has-image flag and the image age.
  • The model reads tokens for weeks 0 … cutoff−1. A projection layer maps each token to 64 dimensions, then a week signal, a learned position embedding and a cutoff (horizon) embedding are added.
  • A 2-layer causal transformer encoder (4 heads) processes the sequence. The causal mask and forward-fill mean that no week sees later captions or images.
  • The last token, together with the horizon embedding, goes to a small regression head.
  • During training, the loss is averaged over all cutoffs (4, 8, 12, 16, 20, 22) for each batch, so one checkpoint serves every cutoff.

Features

  • Text and image embeddings are concatenated per week.
  • Inference starts from raw captions and UAS images: the scripts run Qwen3 and DINOv3, so no precomputed embeddings are needed.
  • Missing weeks are handled with image forward-fill, a has-image flag and image age.
  • One checkpoint predicts at any cutoff week from 1 to 22.
  • The model is a standard Hugging Face PreTrainedModel and loads with AutoModel.from_pretrained(..., trust_remote_code=True).
  • Single-sample and batch inference scripts write JSON (and optionally CSV) outputs.

Repository Structure

config.json                        model config, including normalization statistics
model.safetensors                  trained weights
configuration_multimodal.py        HF config class (loaded by trust_remote_code)
modeling_multimodal.py             HF model class (loaded by trust_remote_code)
multimodal_concat_tf.py            temporal transformer module (loaded by trust_remote_code)
train_args.json                    arguments used to train the released checkpoint
hf/                                same model code, imported by the local scripts
data/embeddings.py                 Qwen3 / DINOv3 embedding and input building
data/multimodal_dataset.py         weekly image placement, dataset, split, normalization
scripts/inference_single_hf.py     single-sample inference
scripts/inference_batch_hf.py      batch inference (JSONL)
scripts/train_multimodal_hf.py     training
examples/                          sample_input.json, sample_batch.jsonl
multimodal_yield_estimation.slurm  SLURM example
component-info.yaml                ICICLE component catalog metadata
requirements.txt
LICENSE

Training Details

  • Split: by field. One farm_field is held out (--test_field, default: the last field in alphabetical order) and its plots are split 50/50 into validation and test; all other fields are used for training.
  • Loss: MSE on standardized yield, averaged over all training cutoffs for each batch.
  • Defaults: AdamW, lr 3e-4, weight decay 1e-2, batch size 16, up to 300 epochs, early stopping on mean validation RMSE with patience 30.
  • Checkpoint: the epoch with the best validation RMSE is saved in Hugging Face format to --out_dir.

Evaluation

The training script evaluates the best checkpoint on the validation and test plots at every cutoff, and writes:

  • metrics.json: validation metrics at the best epoch
  • test_metrics.json: validation and test RMSE, MAE, bias and R² per cutoff
  • test_predictions.csv: per-plot predictions for the test plots
Downloads last month
10
Safetensors
Model size
198k params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for ICICLE-AI/multimodal-yield-estimation

Finetuned
(586)
this model