RAWcooked
Data is never RAW.
Data is always cooked in a way or another.

Jérôme Martinez
No Time To Wait, London, October 2018

Raw ﬁlesHuge size
(4K+ is coming, can be 100 MB/frame, several TB per hour)
1 ﬁle per frame (thousands of ﬁles in a directory)
Not playable as is by several players (VLC...)So many DPX or TIFF format ﬂavors(interoperability issues)

FFV1
Lossless video compression format
Open source, patent free
Adopted by several archives
Being standardized (IETF)
Frames are divided by slices, with checksumsCompression
Example with 1 second at 24 fps 10-bit HD ﬁlm on a 6-core(12-thread) Skylake-X CPU:
24 DPX ﬁles (or in ZIP/TAR uncompressed): 189 MB
1 compressed ZIP ﬁle: 175 MB in 10 seconds
1 compressed LZMA2 ﬁle: 154 MB in 30 seconds
1 compressed LZMA2 solid (don't do that!) ﬁle: 144 MB
1 FFV1/MKV Intra 16-slice ﬁle: 105 MB in 1.5 seconds

Tests on SD 10-bit content
Sponsored by VIAA
On several TB of MXF/JP2k/PCM content
Diﬀerent sources and kinds of material
E5-2698V3 (16 cores+HT, 2.3-3.6 GHz)

Tests on SD 10-bit content

This is an average, results varies depending on thecontent of ﬁlesFrom 0.4x to 2.4x (average 0.7x) real time/thread(encoding/decoding)
From 0.7x to 16x (average 3.5x) the speed of JP2k(FFmpeg)
Be er compression ratio by 2% compared to JP2k
(9% with additional FLAC audio compression)h ps://viaa.be/ﬁles/a achments/.669/VIAA_Preservation_R

Advantages
Open source and free (FFmpeg)
Package is 1.5-3x smaller than DPX/TIFF
Cheksum by "Cluster" (usually 1 second) at containerlevel
Cheksum by "Slice" (you choose how many per frame)
at video level
Files are natively playable by lot of players (VLC,FFplay...)Disadvantages
You lose some metadata(DPX/TIFF header: scan software, some colorimetry
info, ﬁlm type, DPX time code, shu er angle, gamma...)Complicated command line
But...

RAWcookedEasy: just a short command line
"rawcooked YourDirectoryName"Store DPX/TIFF headers/footers in a speciﬁc Matroskaa achment
Store other sidecar ﬁles as Matroksa a achments
Output is a single Matroska/FFV1/FLAC ﬁleEncoding is reversible (bit-by-bit to original ﬁles)"rawcooked YourMatroskaFileName.mkv"

Usages
Storage
Save HDD/LTO space (less €...)Transport
Encode, transport, decode; you save bandwidth (€...
and transfer speed) without changing something else inyour workﬂow (same ﬁles after revert to DPX/TIFF)

Remaining drawbacks
More CPU intensive than uncompressed ZIP(future project: use GPU and/or SSE/AVX)
You lose a complete slice content if there is a single
corrupted bit(Mitigated with several slices per frame, MediaConchcan ﬁx a bitﬂip error and could be extended to a byteerror)

Development
Reto.ch commited to main sponsorship end of 2017
Other sponsors showed interest begin of 2018
First Beta releases ﬁrst half of 2018
Sponsors (and some others) tested with their ﬁles during
the past 6 months
Flagged as stable 2 days ago

Supported input formats
DPX/Raw: 8/10/12/16 bit, RGB/RGBA
TIFF/Raw: 16 bit, RGB
WAV/PCM: 16/24 bit, 1/2/6 channel, 44/48/96 kHz
AIFF/PCM: 16/24 bit, 1/2/6 channel, 44/48/96 kHz
Based on ﬁles from our sponsors
More formats or format ﬂavors on request

Our ﬁrst sponsors
AV Preservation by reto.ch (main sponsor)
National Audiovisual Centre Luxembourg (CNA)
National Library of Norway
Irish Film Institute (IFI)
Northwest University Library
National Library of WalesFinancial sustainability
Open source code provided without lock to sponsors
Deliveries on our website are with a lockDPX 8/10 bit RGB & WAV 2ch 48kHz ﬂavors are usableby default
We provide a key for other format ﬂavors and features(temporary key possible)
1000 € for ﬁrst ﬂavor/feature+ 500 € per additional ﬂavor/feature
To be compared with storage cost saving(cost divided by 2, speed multiplied by 2)

We need you!
Can be improved only if YOU help
Speed improvement (GPU/SSE/AVX)?
Error correction code?
Support of reels?
CFA/Bayer/RGGB support?
Graphical interface?
More input formats?

We need you!
It is not only about money
Tests
Documentation
Advocacy
Use it :)...

QC Workshop
In Luxembourg on November 29
With support from National Audiovisual Centre (CNA)
Lead by Ashley Blewer
Focus on QCTools, MediaInfo, MediaConch, RAWcookedStay in touch
MediaArea: h ps://mediaarea.net, @MediaArea_net
RAWcooked: h ps://MediaArea.net/RAWcooked
Jérôme Martinez: jerome@mediaarea.net
Slides: h ps://mediaarea.net/Events
License (except images): CC BY