Bytes Are All You Need: Transformers Operating Directly On File Bytes
Authors: Maxwell Horton
Sachin Mehta
Ali Farhadi
Mohammad Rastegari
Venue: Accepted by TMLR ()

Abstract: Modern deep learning approaches usually utilize modality-specific processing. For example, the most common deep learning approach to image classification involves decoding image file bytes into an RGB tensor which is passed into a neural network. Instead, we investigate modality-independent representation learning by performing classification directly on file bytes, without the need for decoding files at inference time. This enables models to operate on various modalities without any hand-designed, modality-specific processing. Our model, ByteFormer, improves ImageNet Top-1 classification accuracy by $5\%$ (from $72.2\%$ to $77.33\%$) relative to DeIT models of similar size. Compared to Perceiver IO, our model requires absolutely no modality-specific processing at inference time, and uses an order of magnitude fewer parameters at equivalent accuracy on ImageNet. We demonstrate that the same ByteFormer architecture can perform audio classification without modifications or modality-specific preprocessing. We achieve $95.42\%$ classification accuracy on the Speech Commands V2 dataset (comparable to the state-of-the-art accuracy of $98.7\%$). Additionally, we demonstrate that ByteFormer can operate jointly on images and audio, handling joint classification without explicit knowledge of the input modality. We release our code at https://github.com/apple/corenet/tree/main/projects/byteformer.

Published in Transactions on Machine Learning Research (06/2024)
Bytes Are All You Need: Transformers Operating Directly On File Bytes
Maxwell Horton
mchorton@apple.com
Sachin Mehta
Apple
Ali Farhadi'
Allen Institute for Artificial Intellgence
Mohammad Rastegari'
Meta Al
Reviewed on OpenReview:
https: // openreview. net/ forum? id= RkaqqaA O
Abstract
Modern deep learning approaches usually utilize pondality-speciffe processing. For example, the most common deep lcarning approach to image classification involves decoding image file bytes into an RGB lensor which passed into a neural network. Instead, we investi- gate medaaitty-iddppedded
representation learning by performing classification directly on file bytes, without the need for decoding files at inference time. This enables models to op- erate on various modalities without any hand-designed, modality-speciifi processing. Our model, ByteFormer, improves ImageNet Top-1 classification accuracy by 5% (from 72.2% to 77.33%) relative to DelT models of similar size. Compared to Perceiver 10, our model requires absolutely no modality-specific processing at inference time, and uses order of magnitude fewer parameters at equivalent accuracy on ImageNet. We demonstrate that the same ByteFormer architecture can perform audio classification without modifications or mopelity-specific preprocessing. We achieve 95.42% classification accuracy on the Speech Commands V2 dataset (comparable to the state-of-the-ars accuracy of 98.7%). Addition- ally, we demonstrate that ByteFormer can operate jointly on images and audio, handling joint classification without explicit knowledge of the input modality. We release our code at https: thub.
1 Introduction
Deep learning inference usually involves modality-specific processing. For example, Vision Transformers (VilIs; (Dosovitskiy et al.) 2020)) explicitly model the 2D spatial structure of images by encoding image patches into vectors. Similarly, audio inference often involves computing spectral features (such MFCCs [Lyons] 2009)) to pass into network (Gong et al. 2021 Kim et al. 2022). When a user wants to perform inference file stored disk (c.g. JPEG image file or MP3 audio file), the user must first decode the file into a modality-specificc representation. Fig. la depicts this process for images.
The practice of decoding inputs into a modality-speciffe representation requires hand-crafting an input representation and a model stem for each input modality. Recent works such as Perceiver IO (Jaogle et al. 2021a) and UnifiedIO (Lu al. 2022) have shown that Transformer (Vaswani al. 20077) backbones can be used for variety of different tasks. However, these methods still require modality-specific input
Work done while employed at Apple.

---

Published in Transactions on Machine Learning Research (06/2024)
Embedding
Embedding
Patch emboddings
Byte embeddings
Extract patches
RGB Image
Transformer
Transformer
File decoding
Lineur
Linear
OxFF
0x01
0x8A
OXFF
0x01
0x8A
Input (File bytes)
Output
Input (File bytes)
Oulput
(a) VIT
(b) ByteFormer
ByteFormer vs. VIT. (a) standard vision Transformer (ViT) decodes file bytes into an RGB image. Subsequently, the image split into patches and patch embeddings are extracted and fed Transformer obtain contextualized patch embeddings, which are then classified using a lincar classifier. (b) ByteFormer directly operations file bytes.
Figure
Model
Data format
[[S]
E[L+]
Top-1
VIT
RCB Tensor
3 224 224
196
72.20
VIT"
RGB Tensor
224 224
196
74..5
NWWC
150528
9407
77.06
Model
M
Top-1
See
(M)
(B)
Im/s
(CHW
150528
9407
74.65
Perceiver
67.60
55.9
62.3
BF-Ti (Ours)
TIFF
150668
9415
77.33
Perceiver 10
72.70
62.3
407
PNG
150864
9428
74.94
JPEG
48564
12140
65.92
BF-Ti
/
77.27
1314
8.8
23.74
373
(b)
Table 1: (a) ImageNet Top-1 accuracy of ByteFormer Tiny (BF-Ti) using various file encodings, compared to
ViT. [E[S] denotes the input shape, and [[Le] denotes the token length passed to the Transformer backbone. (") denotes our implementation of ViT. (b) Comparison of ImageNet Top-1 accuracy with Perceiver. M: whether the model accepts various modalities (V: Yes, but with modality-specific modeling. V: Yes). Sec: Train epoch time (not reported for Perceiver to avoid hardware differences impacting results). P (M): Number of parameters (millions). F (B): Number of flops (billions). Im/s: Throughput (images/sec) on an A100 80GB Nvidia GPU, s_" means "not reported".
preprocessing. For instance, Perceiver 10 decodes image files and reshapes them before passing them into the network. Other input modalities processed into different forms.
We hypothesize that it's possible to remove all modality-specific input preprocessing by performing inference directly on file bytes. To test this hesothesis, we develop a Transformer architecture able to operate directly on file bytes. One of the main challenges in operating directly on file bytes is the long token lengths involved. For example, an uncompressed 224 224 TIFF image stored on disk requires 150668 bytes. To flexibly handle long sequence lengths, we replace the multi-head attention the Transformer with shifted window attention (Liu et al. 2021), we add token downsampling layers, and we add convolutional downsampling. We call model ByteFormer (Fig. lb
We demonstrate the efficacy of ByteFormer on ImageNet [Deng et al. 2009 classification, achieving 77.33% accuracy on files stored in the TIFF format (Tab. Ta]. Our model's 's backbone is comparable to ViT-Ti (Touvron et al. 2020 Dosovitskiy et al. 2020) (which achieves 72.2% accuracy on RGB inputs). Compared Perceiver TO (Jacgle al. (021a),, Our method achieves higher accuracy (77.27% vs 72.70%) with an order of magnitude fewer parameters (8.8 million vs 62.3 million; Tab. Ibp. Our model requires absolutely no model-sppeific processing inference time (though we do modality-specíffe data augmentation during
2

---

Published in Transactions on Machine Learning Research (06/2024)
training, see Sec. 4.3 We also present results on PNG files. Surprisingly, our method is even able to operate JPEG files, which include complicated compressions like Hulfman Codes that aren't byte-aligned.
We demonstrate that our classification model can achieve 95.8% accuracy on Speech Commands V2 (War- den 2018), comparable to state-of-the-art (98.7%) (Kim et al.) 2021), without any architecture changes or hyperparameter luning. We use the same training configuration as for ImageNet, demonstrating the general- ity of our method. Additionally, a single model can be trained to perform classification of both images and audio, without architecture changes or hyperparameter tuning.
We also investigate multiple file encodings in our image classification and audio classification tasks. Note that using file bytes as our imput representation means that our model's performance is dependent on the encoding scheme used. Some encoding schemes more naturally represent data from a given domain than other encoding schemes. For instance, uncompressed TIFF images directly store RGB values, whereas compressod formats such as JPEG store bytes without an easily interpreted semantic mcaning. For this reason, we investigate multiple file encodings in our image classification and audio classification lasks to better understand how the chosen encoding scheme impacts performance.
Finally, we perform analyses to understand what patterns ByteFormer learns to recognize. We analyze the learned token embeddings produced by our training procedure, finding that neighboring byte values exhibit strong similarity in uncompressed encodings. From analyzing positional embeddings, we observe that headers are weakly correlated with the image contents for most encodings. We also study the impact of byte ordering on our model's accuracy, finding that encodings that maintain locality produce a higher accuracy.
In summary, our contributions are: (1) To the best of our knowledge, we the first to explore models that directly consume file bytes without requiring modality-speciffe processing at inference time. We call our model ByteFormer. (2) We show that ByteFormer achieves strong performance on a variety of image and audio file encodings without the need for architectural changes or hyperparameter tuning.
(3) We analyze the embeddings lcarned by ByteFormer and study the impact of byte ordering on ByteFormer's accuracy. (4) We release our code al hhttp:: httphh/wwww..pppooooo////////t/a
2 Related Work
To the best of our knowledge, previous works use modality-specific modeling in architecture design. By contrast, ByteFormer does not contain any modality-ppecific modeling. However, our work draws inspiration from previous works, which we discuss in this section.
Architectures With Multimodal Inputs: A few methods have explored the idea of feeding different input modalitics into the same network for procossing. Perceiver IO demonstrates that a Transformer architecture with cross-attention input can be used for a variely of different tasks. However, Perceiver IO decodes file bytes into modality-specific input before feeding into the model. Other recent. works explore using Transformers to process multiple modalities (Yu et al., 2023 Lu al. 2022, Radford et al., 2021 Liu et al. 2023), but also require modality-specifie processing. In contrast, ByteFormer operates file bytes directly, and does not require todality-specific processing at inference time.
Efficient Attention Computation: Recent works have explored efficiently handling long sequence lengths in Transformers. These works are applicable to modeling file bytes directly, since the files we consider can have up to 150, 528 bytes. Due to the O(n²) dependency on sequence length of the attention calculation, many previous works have suggested modifications to the attention computation to improve efficiency. We experiment primarily with shifted window attention (Liu et al. 2021 Bellagy et al. 2020), which uses fixed- size windows to compute attention. We also explore bag attention (Mehta et al. 2020 Chen et al. 2022 in our ablation study. Bag attention computes attention hicrarchically over windows.
Previous works have explored using alternale input represen- tations for images. Gueguen et al.) (2018) perform partial JPEG decoding, stopping when Discrete Cosine Transform Marrhhall (001) cocfficients are formed. They modify ResNet (He et al.) 2015) to ingest this new representation. In (Park & Johnson| 2023), a similar method is used with Transformers. Our work differs in that perform no decoding of file bytes at inference time.
Alternate Image Input Representations:
3

---

Published in Transactions on Machine Learning Research (06/2024)
Analyzing File Bytes: Our method avoids modality-speciffc preprocessing by directly operating file bytes. Directly analyzing file bytes is a technique commonly used in binary analysis, in which computer programs are analyzed for malware content or security issues. See Heena (2021) for an overview. Our work differs in that our primary application is not analyzing the security of computer programs. Our primary appliation is image and audio classification using a machine learning model.
3 Method
We describe our architecture and implementation below. We follow the vision Transformer of Dosovitskiy et al. 2020), with few modifications to handle long sequence lengths.
3.1 ByteFormer
An overview of our model is given in Fig. 16 As input, our model takes in sequence of file bytes, each of which can take on one of 2 values. We simply treat file bytes as token imputs. The first step of our model is to use a learned byte embedding E²a xd to convert file bytes into embeddings of size This differs from ViT's approach (Fig. La). which involves decoding file bytes into an image, converting it into patches, and subsequently generating patch emboddings.
Given the large sequence lengths involved when processing file bytes (which extend up to 150, 528 in experiments), employ strided ID convolution after generating byte embeddings. This reduces downstream compute and memory usage by reducing the sequence length. Our intuition for choosing strided convolution is that neighboring file bytes often contain related information ion.
Next, we add positional embeddings to the resulting embeddings and pass these embeddings to a Trans- former. Note that the cost of self-attention in Transformers is quadratic with respect to sequence length. To compensate for long sequence length and allow our model to learn hierarchical representations more easily, we make two changes following Swin Transformer (Liu et al. 2021
The first change is we replace self-atlention in Transformers with shifted window attention. Unlike Swin Transformer, our imputs are only 1-dimensional. Thus, our windowing and shifting operations only occur over one dimension. This makes our attention mechanism similar to sliding window attention (Beltagy et al. 2020), but with shifting added.
The second change we make our Transformer allow it handle long sequence lengths is we add down-sampling layers to halve the sequence length. The resulting contextualized byte embeddings then averaged and fed to linear layer to produce logits.
3.2 Implementation Details
We follow ViT implementation of Touvron et al. (2020). We set model dimension d = 192 and use 12 layers of Transformers for learning contextualized byte emboddings. We adopt strided convolution with a kernel size of & = 32, and our stride is k/2. We chose these settings as they performed well on TIFF images, and we maintained these settings for other experiments. For JPEG, we perform an ablation and find that reducing the kernel size (and stride) improves performance (Tab. D likely due to the shorter sequence length obtained by JPEG images. Downsampling layers in ByteFormer appear after Transformer blocks 1, 3, 5, 7, and 9. Our window size for windowed attention is w 128.
4 Experimental Setup
When performing inference with a standard Transformer model, the choice of file encoding is irrelevant. For example, doesn't matter whether an image is stored as a JPEG or PNG file because images are decoded into an RGB tensor before inference. By contrast, ByteFormer performs inference on file bytes. To illustrate the ability of ByteFormer to perform modality-indeeeeee with two different input modalities (images and audio) and multiple file encodings (TIFF, PNG, and JPFG
representation learning, we perform experiments
4

---

Published in Transactions on Machine Learning Research (06/2024)
for images, and WAV and MP3 for audio). This section provides overview of these file encodings. Note that file encodings lypically contain a large number of optional settings that influence the resulting file bytes. We use default settings provided by PIL (Clark 2015) or scipy (Virtanen et al. 2020) software packages unless otherwise stated.
4.1 Image File Encodings
We use "fHWC" as an abbreviation for "flattened tensors in height, width, channel order." It refers to uint8 image bytes stored in HWC order without any file headers. is not common store images in this way, since they cannot be decoded without, pre-existing knowledge of their height and width. This serves bascline that demonstrates the performance of ByteFormer rasterized image without file headers.
[HWC:
fCHW: This format is similar to fHWC, but images are stored in "CHW" order.
TIFF: The TIFF file encoding Parsons & Rafferty 1998) allows for many custom configurations. For our experimentation we use the default settings provided by PIL, which do not include compression. This results in a format similar IHWC, but with the addition TIFF image headers describing configuration options and the image size. Comparing our results on TIFF images to fHWC results helps assess ByteFormer's ability to ignore irrelevant file headers.
PNG: The PNG format. (Boutell 1997 contains headers describing PNG configuration options, followed by rows of image data stored in "IDAT" chunks. Each IDAT chunk contains a byte describing the filtering method used for that row's data. The filtering method applies an offset to the row's data based on neighboring pixel values. Thus, our PNG file contains rows of RGB data, with offsets applied, interrupted by occasional bytes that contain file encoding settings. We do not use the optional zlib compression that PNG allows. We expect PNG files to provide more challenges to ByteFormer than TIFF, since image contents are encoded offisess and interspersed with encoding information.
JPEG: JPEG (Wikipodia, 2023) encodes images by applying a series of transformations to compress the im- age before serialization. The RGB image is converted to YCbCr, then downsampled in the chroma channels, then passed through a Discrete Cosine Transform (Marshall 2001), then quantized using coefficietss deter- mined by the JPEG quality factor. The quality factor determines the level of compression, with 100 denoting no compression due to quantization, and lower values indicating stronger compression. After quantization, the coefficients encoded via run-length encoding, followed by Huffman encoding (Raghunathan 2017). Note that Huffman codes not byte-aligned, e.g. they can cross byte boundaries. We expect this to make our modeling task more difficult.
4.2 Audio File Encodings
WAV: The WAV file encoding (Kabal 2022) stores audio signals represented as a sequence of amplitudes. We use single-channel (mono) audio files. The most common configuration options are the bit depth and the frequency. The bit depth corresponds to the precision with which amplitude values are stored. We experiment with a varioty of bit depths, storing audio with 8-bit unsigned integers, 16-bit integers, 32-bit integers, and 32-bit floating-point values. The frequency corresponds to how often amplitude values are chosen. We use 16 kHz, standard choice for audio (Warden 2018
MP3: MP3 Nilsson 2000) uses a perceptual compression method that removes portions of audio that are difficult for humans to detect. The remaining portions are recorded in frequency space. An MP3 file contains a series of frames. Each frame contains a header with encoding settings, followed by the encoded signal in frequency space. We use standard scttings for MP3 provided by the pydub (Robert et al., (018)) software package. We expect MP3 encodings be more difficult to handle than WAV files due the compression applied.
5

---

Published in Transactions on Machine Learning Research (06/2024)
4.3 Preprocessing
Some file encodings such as TIFF and MP3 are not frequently used in machine learning datasets. To allow for comparisons on single dataset across a variety of file encodings, we must re-encode files with different file encodings.
At training time, we decode the file (e.g. read the contents into an RGB tensor in the case of images, or read the contents into 1D tensor in the case of audio), then perform standard training augmentation (e.g. random cropping in the case of images, or temporal augmentation in the case of audio), then save the result in the desired file encoding. We find that standard training augmentation is important for model accuracy, Thus, our training method is implicitly dependent on the input modalily due lo our augmentation.
At inference time, we do not need knowledge of the input modality. We only need to ensure that our model inputs use the correct file encoding. For example, for TIFF experiments on ImageNet, we procompute 224 x 224 crops of the validation images and save them in the TIFF format. Such preprocessing only necessary because the ImageNet validation is not already stored in the desired format. Similarly, for audio classification, re-cncode the audio clips in Speech Commands V2 into the desired format before validation.
5 Evaluating ByteFormer
One notable advantage of learning representations using file bytes is the potential for single model to be applied seamlessly across various input modalities, thereby eliminating the need for modality-specific modeling. However, it's crucial to assess the potential trade-off in accuracy associated with modality- independent learning. In this section, we empirically evaluate the capabilities of ByteFormer.
We begin by evaluating ByteFormer on the ImageNet dataset Deng et al. 2009 demonstrating that it achieves comparable or superior performance compared VIT (Sec. 5.1 Additionally, we assess ByteFormer on the Speech Commands dataset (Warden 2018), showcasing its competitive performance against state-of- the-art methods in audio classification (Sec. 5.2
Furthermore, we extend unimodal evaluations in both images and audio to multimodal setting. By training a single classifier capable of classifying both images and audio directly from bytes, we show that the resulting model maintains competitiveness with unimodal counterparts (Sec. 5.3).
5.1 Evaluating ByteFormer on ImageNet
Dataset and training details. We evaluate ByteFormer on 1000-way classification on ImageNet. Our primary comparison with VIT (Touvron al.) 2020),, since our Transformer backbone's size parameters match it. We refer to this architecture as ViT-Ti to emphasize that the distillation in Touvron et al. (2020)
is not used. We refer to our architecture as BF-Ti to highlight the fact that our hyperparameters match the "tiny" variant of the Transformer. In spite of our inclusion of shifted window attention in our architecture, we use ViT as our primary baseline rather than Swin since the smallest Swin Transformer Liu et al. (2021) has over 3x the parameter count our largest model. See Appendix for comparisons with Swin Transformer.
We train using CVNets (Mehta et al., 2022). For ImageNet, we use a batch size of 48 on single machine equipped with NVIDIA A100 GPUs At training time, we use random resized cropping, random horizontal flipping, RandAngment (Cubuk et al.)
2019), and RandomErase (Zhong et al.) 2017) before storing the image in the desired file encoding (Sec. 4.3) We train with AdamW (Loshchilov & Hutter 2017) with weight decay 0.05, and a cosine learning rate schedule that anneals the learning rate from 0.001 to 0.00002, with 7500 warmup iterations.
For ImageNet experiments, we report Top-1 accuracy of models trained with exponential moving average CCai et al. 2021) of weights with momentum 0.0001, which on average increased accuracy by roughly 0.25%.

---

Published in Transactions on Machine Learning Research (06/2024)
Model
Input
w
[[S]]
Top-1 1
BC- ResNet-8
log Mel
40 x 98
98.70
128
32
64058
95.80
BF-Ti (Ours)
W-FP32
q
E[S]
Top-1
128
64058
95.51
128
32
64044
94.90
100
128
32
48564
60.86
BF-Ti (Ours)
W-INT32
128
64044
95.27
100
128
16
48564
64.86
128
32044
94.81
100
128
48564
65.92
BF-Ti (Ours)
W-INT16
128
16
32044
95.51
60
128
32
8436
31.80
128
32044
95.13
60
128
16
8436
50.11
128
32
16044
92.28
60
128
8436
56.26
128
16044
94.39
BF-Ti (Ours)
W-UINTS
60
128
8436
62.52
128
16044
94.81
128
16044
93.99
60
32
32
8436
37.22
128
3465
88.39
60
32
16
8436
50.24
128
3465
88.00
60
32
8436
56.74
BF-Ti (Ours)
MP3
32
3465
88.69
60
8436
59.52
32
3465
89.19
(a)
(b)
Table 2: (a) ImageNet Top-1 accuracy for BytePormer Tiny (BF-Ti) for different JPEG quality factors g. window sizes w, and convolutional kernel sizes k. E[S] denotes the expected shape of the inputs during validation. (b) Results for audio classification with BF-Ii on the Speech Commands V2 dataset. "W-" denotes WAV files with the given bit width. E[S] denotes the shape of network inputs.
Effect of image file encodings. Tab. La summarizes results for variety of file encodings on the ImageNet dataset. For BF-Ti, we use window size 128 and convolution kernel size k = 32 for all models except JPEG, for which we find & 8 to perform better. Our method surpasses VIT accuracies for TIFF, PNG, fCHW, and fHWC encodings. We note that fHWC outputferforms fCHW by 2.41%. This indicates that channel-wise locality preserves accuracy better than spatial locality.
Notably, our results for all encodings except JPEG surpass the modality-speciffe baseline. This likely a result of using a higher parameter count. for our embedding layers, and using a longer token length (which requires more computation). More analysis of runtime characteristics compared to modality-seecific models appcars in Appendix A We emphasize that our focus is not to obtain superior cfficieney-accuran tradc-off compared to modality-speciffe models. Our focus is on analyzing the feasibility of avoiding mopelit processing by using file bytes as inputs.
We find training on JPEG to be more difficult than other modalities. This is due to the highly nonlincar and variable-length JPEG encoding. Note that, since the Huffman coding scheme used in JPEG is not byte-aligned, a byte value's "meaning" highly dependent on the neighboring bytes. This in contrast with other encodings like TIFF, in which a byte value's meaning is independent of neighboring bytes (for example, OxFF always corresponds to bright pixel-channel, regardless of neighboring byte values). When using JPEG, our byte embedding presents a challenge since particular byte value will always be projected to the same embedding regardless of the value of neighboring bytes. Even so, our model is able to obtain 65.92% accuracy.
Effect of k. We investigate the influence of our model's kernel size k on JPEG accuracy in Tab. 2a We find that reducing & from its default value 32 increases accuracy. Since JPEG images have a smaller token length than TIFF or PNG, they are likely less compressible. To further explore this, we investigate two settings for JPEG quality factor in Tab. Pa We find that lower quality factors result in lower token lengths, thus reducing & improves accuracy. We also try reducing w, but accuracy does not improve.
7

---

Published in Transactions on Machine Learning Research (06/2024)
Modality
Balanced?
Epochs
IN
SC2
Image
Audio
300
77.33
300
95.80
X
300
77.47
85.71
/
300
76.64
90.08
150
75.46
89.81
Table 3: Joint image and audio classification compared to unimodal classification with ByteFormer. Bal- anced?: Whether the SC2 datasct replicated 33x to achieve a balanced dataset in the multimodal case. Epochs: The number of training epochs. IN: The top-1 on 1000-way image classification. SC2: The top-1 12-way audio classification.
Comparison with existing multimodal methods. We compare our work with ViT in Table la Byte-
Former can surpass the accuracy of ViT when operating on uncompressed files. Note that ViT uses modality- specific preprocessing, and can only be run on images. We compare our work with Perceiver 10 in Table 1b. Our work requires no modality-specific preprocessing, whercas Perceiver requires pixel buffers to be docoded and reordered. Our method uses an order of magnitude fewer flops and parameters than Perceiver (Jaegle et al.] 2021b a). We present more details of our method's computational efficiency compared to related works in Appendix [ Note that previous works include modality-speciiicc processing, which disadvantages our model in the comparisons.
5.2 Evaluating ByteFormer Speech Commands V2
Dataset and training details. We evaluate ByteFormer on 12-way audio keyword classification (including "background" and "unknown" classes) of 1-second audio clips sampled at 16 khz using the Speech Commands V2 dataset. Similar to Kim et al. (2021). we train our model with MixUp (Zhang et al.) (017), noise augmentation, and time shifting augmentation. Then, we store the audio in the desired format. We use the same training and architecture apper-parameters as ImageNet (Sec. 5.1 to demonstrate that our method does not require modality-speciff hyperparameter tuning. For Speech Commands V2, we found FMA to sometimes increase and sometimes decrease accuracy, so omit it. We train models 4 NVidia A100 GPU machines.
Effect of audio file encodings. Tab. 2b summarizes results for variety of file encodings on the Speech Commands V2 dataset. BF-Ti achieves accuracies of up to 95.51% on FP32 WAV files. On MP3 files, accuracy reduced. We believe the compression used in the MP3 format makes the lcarning task more difficult. This analogous to JPEG compression reducing accuracy on image classification.
Effect of k. We investigate the influence of convolutional kernel size on model accuracy in Tab. 2b general, the optimal & decreases when the expected number of input tokens decreases. This matches observations in ImageNet JPEG experiments. For MP3 files, we observed that k 32 resulted in unstable models due to the drastic reduction in token length. For MP3, we additionally experiment with window size w 32, but it does not improve results.
Comparison with existing methods. Results for audio classification are given in Tab. 2b BF-Ti achieves accuracies of up to 95.51% on WAV files, comparable to the state-of-the-art method BC- NesRes [Kim et al. (021). Note that BC-ResNet-8 performed no parameter tuning relative our ImageNet training recipe (besides ablating choices of and k). Our best-performing model has window size 128 and kernel size k 32.
specifically designed for audio processing. By contrast, we
8

---

Published in Transactions on Machine Learning Research (06/2024)
0
2
1
3
0
11
B22
B3
B44
0
0
1
1
B2
Attention
Top- 1
2
2
Window
77.33
Bag
3
3
144
B3
B4
(a) Full Attention
(b) Window Attention
(c) Bag Attention
(d) Results
Figure 2: (a-c): Illustaaioon of the types of attention used in ablations. Bag attention is computed in two stages. First, individual bags compute allention. Then, attention is computed across bags. (d): ImageNet Top-1 accuracy of BF-Ti with different types of attention. We run out of memory with full attention.
IN TIFF
IN ICHW
IN PNG
IN JPFG
SC FP32
SC UINTS
0
1.0
token
0.8
128
0.6
0.4
positional
128
0.2
0.0
0
128
128
0
128
128
128
0
128
Figure 3: lor /l/([a=]|-1ll]) for pairs wr of token embeddings (Lop row) and positional embeddings (bottom row) learned by BF-Ti. We show results for various file encodings on ImageNet (IN) and Speech Commands V2 (SC).
5.3 Evaluating ByteFormer in the Multimodal Setting
Dataset and training details. We experiment with training single classifier classify both images and audio. To do so, we conjoin the ImageNet and Speech Commands V2 datasets and train single model for 1012-way classification (since ImageNet has 1000 classes and Speech Commands V2 has 12 classes). We optionally balance the sampling between the two datasets to account for differences in the dalasets' nunbers of samples by replicating Speech Commands V2 by 33x. Because this doubles the training time relative to ImageNet training, we additionally experiment with reducing the number of epochs from 300 to 150. Note that there is no aligument between the classes represented by these two datasets.
Comparison with unimodal ByteFormer. We compare our multimodal models with our unimodal models in Tab. 3 Without dataset balancing, our method produces accuracies on par with the ImageNet TIFF models, but with a lowered accuracy Speech Commands V2 with WAV-FP32 files. Adding dataset balancing improves SC2 Top-1 results at the expense of minor reduction in ImageNet accuracy. Note that Speech Commands V2 suffers from overfitting in the balanced dataset case. Accuracy drops from 95.68% (hallway through training) to 90.08% at the end of training.
6 Analysis
Ablating alternate attention methods. We study three state-of-the-art sele-attention methods for han- dling long sequence lengths Transformers: (1) full self-attention (Vaswani al.) 2017 Dosovitskiy al.
2020 et al. (022)) where each token attends on every other token, (2) shifted window attention (Liu al.
2021
Beltagy et al. 2020) where tokens divided into local windows and each local window computes
9

---

Published in Transactions on Machine Learning Research (06/2024)
Augmentation Top-1
Random Shuffle
Stride 5.64
Window Shuffle
Cyclic 60.97
Reverse 61.23
Baseline 60.81
3.06
18.14
Table 4: Ablation showing the Top-1 ImageNet accuracy of BF-Ti on JPEG images using kernel size h = 32 and quality factor 100. See text for details.
self-attention independently, and (3) bag (or hierarchical) allention Mehta et al. 2020: Chen et al. 2022)
where tokens are broken up into bags and each bag computes intra-bag self-attention. Inter-bag self-attention is then computed on the resultant output. These different methods are visualized in Fig. 2aj while ImageNet results on TIFF imputs are summarized in Fig. 2d We choose TIFF for these experiments because of its long sequence length (Tab. Tab. We find window attention to outperform bag attention. Note that full attention cannot be run due to its (n²)) dependence on sequence length In main experiments, used shifted window attention.
Learned token embeddings. We study the token embeddings learned by ByteFormer. These embeddings are used to project file bytes into vector representations. In Fig. 3 (top row), we observe the absolute value of the cosine distance |x-yy/((| ([|]=||-Ill/|] between each pair of token embeddinggs æ, y on a variely of file encodings. We choose this metric to highlight the difference between correlecree embeddings (bright patches) and uncorrelated embeddings (dark patches). The pattern varies substantially across input encodings and tasks. In TIFF, PNG, and ICHW, we observe a bright band off of the diagonal, corresponding to high correlation between bytes and their neighbors. This matches our expectations, since replacing a byte with its neighbor only slightly alters the image. This does not hold for JPEG due to the Huffman encoding step. We also observe that the correlation between token embeddings in the float32 encoding of Speech Commands V2 is generally weak. We believe this occurs because the float32 audio amplitude value is split. across four bytes in the file encoding, weakening the association between byte values and amplitudes.
Learned position embeddings. We visualize the absolute value of the cosine distance between the first 256 positional embeddings learned by ByteFormer in Fig. 3 (bottom row). For JPFG, we see strong band of highly uncorrelated values at carly positions, corrosponding to the file header. Later positions demonstrate interesting patterns that may arise due to the Huffman encodings crossing byte boundaries. In TIFF, a small band of highly uncorrelated values is visible early on, corresponding to the header (which is shorter than in the JPEG case).
Byte ordering. To better understand ByteFormer's behavior, we ask, does ByleFormer simply learn byle frequencies, or is the byte ordering relevant? In Tab. | apply series of augmentations during train- ing and validation. We focus on the case of JPEG compression at quality factor 100 with our standard kernel size k = 32. Each augmentation modifies the byte order of the inputs in some way. In random shuffle, we randomly reorder the bytes during training and validation. The order is redrawn every it- eration. This severely degrades accuracy. Next, we perform a strided sampling with stride size 1024 (e.g. [0, 1024, 2048. 1, 1025, 2049, .)). This slightly improves accuracy over the previous method by improving byte order consistency. Next, we experiment with window shuffle, in which the bytes from each window of size 1024 are consistently permuted. This increases accuracy to 18.14%. Next we experiment with cyclic shift in which the second half of the image bytes are moved to the beginning. Accuracy matches the baseline (unaltered JPEG bytes) closely. Similarly, reverse, in which the byte order is reversed, preserves locality well and matches the baseline. Our model is sensitive to locality, and does not only learn byte frequencies.
Inference on obfuscated inputs. We provide additional experiments that explore the idea of performing inference on obfuscated inputs in Appendix B
7 Limitations
The accuracy of ByteFormer depends on the file encoding chosen. As shown in Soc. choosing JPEG over TIFF results in a reduction of accuracy on ImageNet. Adding invariance to file encodings is future work. Our focus is analyzing the feasibility of learning directly from file bytes.
10

---

Published in Transactions on Machine Learning Research (06/2024)
Additionally, note that separate network is trained for every file encoding in setup. If inference for different file encoding is desired, the file will need to be re-encoded in the proper format, new network will need to be trained. Given our model's ability to handle multiple modalities in the audiovisual case, we believe a single model could also be trained to handle multiple file encodings. We leave detailed analysis of this question as future work.
Additionally, our method has only been evaluated on classification for images and audio. Experimenting with other domains (video, text) and tasks that require fine-grained localization (detection, segmentation) is exciting future work.
Finally, our method removes modality-specifie preprocessing from inference, but still uses mpdelity-speciffc preprocessing during training in the form modality- data augnentation (Sec. 4.3 Perhaps leveraging modern large-scale datasets would eliminate the need for modality augmentations. We leave this investigation for future work.
8 Conclusion
We analyze the feasibility of learning directly from file bytes. To do so, we present ByteFormer, a model that consumes only bytes and does not explicitly model the input modality at inference time. We show that it achieves competitive performance on image classification. The same architecture achieves competitive performance on audio classification without hyperparameter tuning or modality-specific preprocessing. It also perform multimodal classification of audio and images. In our analysis, we perform ablations of architectural choices used to handle the long sequence lengths inherent in file encodings. We also analyze the learned embeddings and the impact of byte ordering on model accuracy.
Acknowledgments
We would like acknowledge useful discussions with Peter Zatloukal, Dmitry Belenko, Mehrdad Farajtabar, Mackenzie Binns, Robert Karl, Minsik Cho, Fartash Faghri, Yanzi Jin, and Mohammad Sekhavat.
References
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 2020.
T. Boutell. Png (portable network graphics) specification. http://www.ffeeiiiiiorrrrrr////e 1997. Accessed: 2023-03-05.
Zhaowei Cai, Avinash Ravichandran, Subhransu Maji, Charless C. Fowlkes, Zhuowen Tu, and Stefano Soatto. Exponential moving average normalization for self-supervised and semi-supervised learning. 2021 IEEE/CVF Conference on Compuler Vision and Pallern Recognilion (CVPR), pp. 194 203, 2021. URL
attps://api sww.aaiiissmmannccccchlrrrrrr www.aaarrrrrCooooooDDDD 201662114
Richard J Chen, Chengkuan Chen, Yicong Li, Tiffany Y Chen, Andrew D Trister, Rahul Krishnan, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition, pp. 16144- 16155, 2022.
Alex Clark. Pillow (pil fork) documentation, 2015. URL http:://bullmmdaaaarrtthhddeeerd ppd/pp/llow/lates/pplllooo pdff
Ekin Dogus Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V. Le. Randaugment: Practical automated data augmentation with reduced scarch space. 2020 IEEE/CVE Conference on Computer Vision and Pallern Recognition Workshops (CVPRW), pp. 3008 3017, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEFE Conference on Computer Vision and Pattern Recognilion, pp. 248 255, 2009.
11

---

Published in Transactions on Machine Learning Research (06/2024)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ArXiv, abs/2010.11929, 2020.
Yuan Gong, Yu-An Chung, and James R. Glass. Ast: Audio spectrogram transformer. ArXin, abs/2104.01178, 2021.
Lionel Gueguen, Alex Sergeev, Ben Kadlec, Rosanne Liu, and Jason Yosinski. Faster neu- ral networks straight from jpeg. In S. Bengio, H. Wallach, H. Larochelle, Grauman, N. Cesa-Bianchi and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 31. Curran Associates, Inc., 2018. URL https::/prroceedings neurips cccppaper/2018/fl1e pdf
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEFE Conference Computer Vision and Pattern Recognation (CVPR), pp. 770-778, 2015.
Heena. Advances in malware detection- overview, 2021.
Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin lonescu, David Ding, Skanda Koppula, Andrew Brock, Evan Shelhamer, Olivier J. H'enaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver io: A general architecture for structured inputs & outputs. ArXiv, abs/210 22//211077..999 2021a.
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and João Carreira. Per- ceiver: General perception with iterative attention. In International Conference on Machine Learning, 2021b.
Peter Kabal. Wave file specifications. WAVE/WAVE. html 2022. Accessed: 2023-03-05.
http:///wwwmmmppcc.e.... http:///www..........././///////
Byeonggeun Kim, Simyung Chang, Jinkyu Lee, and Dooyong Sung. Broadcasted residual learning for efficient keyword spotting. In Interspeech, 2021.
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 27, 2022, Proceedings, Part IX, pp. 280 296. Springer, 2022.
Hao Liu, Wilson Yan, and Pieter Abbeel. Language quantized autoencoders: Towards unsupervised text- image alignment, 2023. URL https: http////xvvv/.//////////////.atl ab///bbb.//000002222222
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. 2021 IEEE/CVF International Con- ference on Computer Vision (ICCV), pp. 9992-10002, 2021.
Nyy Loshchilov and Frank Hulter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2017.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. ArXiv, abs/2206.08914 2022.
James Lyons,
Mel frequency cepstral htwpyy/cooommmccclanoonssss coefficients-mfoes/ 2009. Accessed: 2023-03-06.
coefficient
(mfcc)
tutorial.
http:/
Dave Marshall. The discrete cosine transform. http:///ueessssccaa..//D.DD... Mawpaaallllnniiiiiiiiiniono html, 2001. Accessed: 2023-03-06.
12

---

Published in Transactions on Machine Learning Research (06/2024)
Sachin Mehta, Ximing Lu, Donald L. Weaver, Joann G. Elmore, Hannanch Hajishirzi, and Linda G. Shapiro. Hatnet: An end-to-end holistic attention network for diagnosis of breast biopsy images. ArXin, abs/2007.13007, 2020.
Sachin Mehta, Farzad Abdolhosseini, and Mohammad Rastegari. Cvnets: High performance library for computer vision. Proceedings of the 30th ACM International Conference on Multimedia, 2022.
M. Nilsson. The audio/mpeg media type. cessod: 2023-03-05.
2000. Ac-
Jeongsoo Park and Justin Johnson. Rgb no more: Minimally-deccdeed jpeg vision transformers. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pallern Recognition (CVPR), pp. 22334 22346, June 2023.
G. Parsons and J. Rafferty. Tag image file format (liff). http:://www...frooooooooootoftfp 1998. Accessed: 2023-03-05.
Alec Radford, Jong Wook Kim, Chris Hallacy Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference mackine learning, pp. 8748 8763. PMLR, 2021.
Vijay Raghunathan. Ece264: Huffman coding, http:///wehhhonioopo//c///////// huffman, 2017. Accessed: 2023-03-07.
James Robert, Mare Webbie, et al. Pydub, 2018 URL htttp:/pyddub... com/
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv'e J' J'egou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, 2020.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. ArXiv, abs/1706.03762, 2017.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Ev- geni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan van der Walt, Matthew Brett, Joshua. Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Fric Jones, Robert Kern, Eric Larson, C J Carey, Îlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio Н. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272. 2020. doi: 10.1035/s41592-D0 019-066---.
Pete Warden. Speech commands: A dataset for limited-vocabularyy speech recognition. abs/1804.03209, 2018
ArXiv,
Wikipedia. Jpeg. http://www.kkpppaaaaa.rg////////
2023. Accessed: 2023-03-05.
L. Yu, Daniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. Megabyte: Predicting million-byte sequences with multiscale transformers. ArXiv, abs/2305.07185, 2023. URL https: 7/api. wwwpaattttstcclaaaaoooorrrreeeee
258676582
Hongvi Zhang, Moustapha Cissé, Yann Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. ArXiv, abs/ 1bs/1710.09412, 2017.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random crasing data augmentation. In AAAI Conference on Artificial Intelligence, 2017.
13

---

Published in Transactions on Machine Learning Research (06/2024)
A
Performance
Our goal is to directly perform inference on file bytes. To our knowledge, all previous methods involve some level of domain-specifie modeling at inference time (including file decoding and different stems for different input domains). Therefore, direct comparison dasadvantages our model. Nevertheless, it's important to characterize the runtime of our model, and compare to previous approaches. This helps to contextualize our model's performance.
We compare BF-Ti to related works in Tab. We show performance ImageNet, including efficiency and accuracy metrics. We only report train time and throughput for models we trained ourselves. This is to avoid hardware differences creating inconsistent results. For these experiments, we tuned BF-Ti's batch sizes to maximize GPU utilization. Note that this improved training time by a relatively small amount (less than 10%) compared to the experiments in Sec.
Our model's size and accuracy (8.82 million parameters, 77.27% falls between ViT-Ti (5.72 million pa- rameters, 78.62%) and VIT-S (22.05 million parameters, 73.20%). Our model's forward pass time is slower due to the large number of tokens being modeled. Domain-specifie modcling (which our model avoids) drastically reduce compute time.
Compared to Perceiver IO, our model achieves a comparable accuracy with an order of magnitude fewer flops and an order of magnitude fewer paramoters. Note that our model performs no domain-specific modeling at inference time. By contrast, Perceiver IO includes domain-specific modeling (file decoding, lensor reshaping, and optionally convolutions) inference time.
Model
M
[[L]]
Top-1
Sec
P PMM)
F (B)
Im/s
MobileNetv3 Large ResNet-50
X
N/A
75.10
5.4
0.22
9615
X
N/A
78.12
25.6
4.02
3488
Swin-T
X 196
81.3
29
4.5
ViT-S p16
X 196
78.62
336
22.1
4.61
3594
VIT-Ti p=16
X 196
73.20
334
5.7
1.26
6885
ViT-Ti p=14
X
256
74.62
331
5.7
1.70
4970
ViT-Ti p=8
X
784
77.44
824
5.7
7.06
1243
RGB No More ViT-Ti
X
196
75.10
5.7
1.26
6885
Perceiver (learned pos)
N/A
67.60
55.9
62.3
Perceiver TO (learned pos)
N/A A
72.70
62.3
407
Perceiver (соnv)
N/A
77.40
42.1
367
Perceiver 10 (conv)
N/A
82.10
48.6
369
BF-Ti k-32
9415
77.27
1314
8.8
23.74
373
BF-Ti k=32-4 -C
9415
74.54
1122
7.6
12.63
370
BF-Ti k-32 -C -NPE
9415
68.42
1121
5.8
12.63
372
BF-Ti k=4 f0.05
/ 3762
67.53
368
6.7
5.70
1687
BF-Ti k=4 f=0.1
/ 7524
71.26
580
7.4
11.07
875
BF-Ti k-8 f-0.25
/ 9407
73.65
769
7.9
15.40
634
Table 5: ImageNet Top-1 accuracy. M: whether the model accepts various modalities (x: No. /: Yes, but with modality-spccific modeling. /: Yes). E[L.]: length of token inputs to transformer (after Conv1D for BF-Ti. Note, Perceiver feeds inputs through cross-altention, so this concept doesn't 1 directly apply). Sec: Train epoch time. P (M): Number parameters (millions). F (B): Number of flops (billions). Im/s: Throughput (images/scc) on an A100 80GB Nvidia GPU. 4_" means "not reported". For ViT, p is patch size. For BF-Ti, the convolution's kernel size, and f indicates fraction of retained pixels for masking camera experiments (Appendix B.2 "-C" indicates replacement of Conv with windowed mean. *NNPE" indicates ablation that removes the positional embedding.
14

---

Published in Transactions on Machine Learning Research (06/2024)
Embeediigg
Embooding
Byte embeddings
Byte emboddings
0x9C
0xSB
0xD2
Transformer
Remapped Input
Lincar
OxFF
0x01
DvKA
Input (File bytos)
Outpat
Masking Camera
Outgut
(a) Byte Permulation
(b) Masking Camera
Figure 4: An overview our byte remapping method and our masking camera method. (a): In our byte remapping method, we remap byte values using permutation function before passing inputs our model. (b): In our masking camera method, our model inputs are heavily masked images rasterized into continuous array.
B Inference on Obfuscated Inputs
ByteFormer capable of performing inference directly on file bytes. In this section, we explore the implica- tions of presenting inputs to ByteFormer in non-standard encodings in an effort to obfuscate the inputs.
For example, consider smart home security system that performs image classification to determine what objects are in the camera's field of view. If an adversary is able access the inputs to the model due to security flaw in the system, the adversary can view the contents of the input to the network.
If a model takes an obfuscated representation of the data, the attack surface that an adversary can leverage could be reduced. There is a natural tension between designing a data representation that supports inference and designing data reprosentation that preserves privacy. Conceptually, any data representation that supports network inference must leak information about the underlying data, and therefore cannot be fully secure.
In spite of this, obfuscating the inpuls to a network can increase the difficulty of an adversary's task, or reduce the amount of information available to an adversary. In the following subsections, we discuss two ideas that leverage the unique propertics of ByteFormer to obfuscate network inputs with minimal loss of accuracy.
We reiterate that these methods alone are not sufficient to provide formal privacy guarantees. We present these ideas to demonstrate that ByteFormer has special properties that be leveraged to help make adversary's job more difficult, and to inspire future work in privacy-preservin inference.
In the following subsections, we discuss two ideas for performing inference on obfuscated representations with BytePormer. In Appendix B.1 we discuss permuting file bytes before passing them to our model (Fig. 4a In Appendix B.2 we discuss a custom camera capture system that passes masked images to our model
(Fig. 4bb.
B.1 Inference Permuted Inputs
ByteFormer is designed to perform inference file byles without converting them into standard input representation (e.g. RGB tensor in the case of images). Inspired by this, demonstrate that ByteFormer can perform inference on images that have been altered in an effort to obfuscate their contents. Specifically, we explore altering the image contents by permuting byte values (Fig. 4a
We start with an observation about a special property of Transformer models. Consider a permutation Φ: (0, 1,2,... 255) + {0, 1,2..... 255). Let 7 denote a token embedding, and let fo denote the subsoquent Transformer. It's easy see that, for a given é, there exists a F--1 such that T =(x). Ta-1 is simply a copy of with embedding vectors reassigned to different indices. Thus, f((o-1(((z)) f(r(x))
15

---

Published in Transactions on Machine Learning Research (06/2024)
None
U[-5,
U[-10, 10]
U[-20, 20]
Model
Noise level
ViT-Ti
BF-Ti
None
51.61
77.39
U|
50.77
77.27
U 10]
49.50
77.17
T| W 20, 20]
43.84
76.31
(b)
Figure 5: (a): A sample image from the ImageNet validation set, with uniform noise applied (top row), and with byte remapping & additionally applied (bottom row). (b): ImageNet Top-1 results for obfuscation with Φ. We show results with no noise, and with uniform noise in [-a, a) added. We use the IHWC encoding.
The implication of this statement is that the network fo can operate on re-encoded inputs 6(x) without requiring any retraining as long as the network's token embodding 7 is reordered to Tg-1. Since our model operates on token embeddings, this re-encoding method is compatible with our model. By contrast, this re-encoding method is not compatible with the patch embedding used in ViT.
We leverage the above observation to evaluate ByteFormer on obfuscated inputs. Before training, we choose permutation @ at random. All training and inference inputs are remapped using 6. More generally, we can use more sophisticated methods for altering input representations by choosing more complicated forms for の. As Byteformer can handle highly nonlinear JPEG encodings, we expect it to perform well on a variety of alternative encodings that an outside observer might not be able to casily guess.
Note that the analysis of the effectiveness of this obfuscation against adversary depends the threat model used. For example, if an adversary has access to large number of encoded samples d(x), analysis of byle statistics might suggest that strings of common bits correspond to patches of blue sky in images. The adversary's task is made more difficult given certain file encodings (e.g. the highly nonlinear JPEG encoding). We do not make strong claims regarding the level of security provided by different choices of ø.
Dataset and training details. We follow the ImageNet training and evaluation setup of Sec. 5.1 using the fHWC file encoding. We add two additional steps to the pipeline.
Our first additional step is, we optionally apply uniform noise to our images. We do this to prevent regions of constant pixel value from appearing in the image, thereby further obfuscating the image. In particular, we add noise from a uniform distribution U|-a, a] sampled from -a to a (inclusive) to each pixel channel independently, then compute the result modulo 256.
Our second additional step is, we apply randomly chosen permutation function to byte values before passing them ByteFormer.
Results. Examples of obfuscated images are shown in Fig. 5a We observe that, without noise, byte remapping retains shape information. A region of the image that is dominated by single pixel value will continue to be dominated by new (remapped) pixel value. This reduces the efficacy of obfuscation, since clear patterns are visible in the data. As shown in Fig. 5a] the upper right corner the image becomes less recognizable as noise from progressively larger ranges used. In Fig. 5b we observe that our method is resilient to this obfuscation, but ViT is not. At the strongest level of noise, our model loses only 1.08% accuracy relative to the setting without noise.
16

---

Published in Transactions on Machine Learning Research (06/2024)
100%
25%
10%
3%
Kept
75%
50%
25%
10%
5%
3%
k
32
16
8
4
Top-1
74.77
75.36
74.04
71.35
68.11
64.15
(a)
(b)
Figure 6: (a): An ImageNet validation image captured by our hypothetical masking camera in which the given fraction of pixel channels are kept. Note that positions of retained pixel channels is discarded by the camera. To make visualization possible, we include the positional information implicitly by placing unmasked pixels in the correct position. (b): ImageNet Top-1 accuracy for our masking camera experiment with BF-Ti when the given fraction of pixel channels are kept.
B.2 Masking Camera
We describe another application of BytePormer to inference with obfuscated inputs (Fig, 4b In this scenario, hypothetical camera captures non-standard, obfuscated representation to allow for inference without building a full RGB image. This custom representation could take a variety of forms. In our experimentation, we consider a hypothetical "masking camera" that masks out a large fraction of its pixel channels. The camera stores the remaining unmasked pixel channels an array without retaining the coordinates of pixel channels on the image sensor. In this scenario, an adversary could not obtaín a high-fidelity reconstruction of the input image. Even if the adversary could guess pixel channel locations, the adversary could only reconstruct low-fidelity image due to the high masking ratio used.
Dataset and training details. We follow the ImageNet training and evaluation setup of Sec. 5. using the fHWC file encoding. To simulate the effect of our hypothetical masking camera, we add one additional step to the pipeline. We precompute a fixed byte mask, and apply the mask to the fCHW bytes. Unmasked bytes are then re-packed into a smaller buffer.
Results. Masked images are shown Fig. 6a
Note that, for case of visualization, we show images before the unmasked bytes are repacked into smaller buffer. This means that the visualization Fig. [6a contains more information than the inputs to ByteFormer, because the byte re-packing destroys spatial information pertaining to where the unmasked pixels are in the original image.
Fig. 6b summarizes our results for our masking camera. For these experiments, we cannol provide ViT-Ti baselines because, unlike ByteFormer, ViT-Ti is not capable of ingesting pixel values without any indication of their placement in the image. This is because of the patch embedding used in ViT-Ti.
At 10% pixel retention, the content of image is hard visually perceive (Fig. 6a). As shown in Fig. 6b our accuracy 10% pixel retention is 71.35% comparable to the original ViT-Ti model operating unmasked images.
Note that this masking technique can be combined with the byte remapping technique (Appendix B.1 to further obfuscate network inputs.
17


Reviews:
[{"review_id": "cv0vlpX2Mn", "reviewer": "Reviewer_uWSf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* The paper is well written and makes for a pleasant read. The contributions are clear, and, in my opinion, satisfy the technical correctness requirements of a TMLR article.\n* The main direction of the work is also interesting, and makes enough progress on the problem of learning from bytes to lay the foundation for future work on this topic. This work is especially timely as transformer architectures become more prevalent for multimodal processing.\n* The experimental setup is clear for the most part, and the findings do enough to convince the reader of the potential promise of learning from raw bytes.\n\n**Weaknesses**\n* Limited numerical results:\n  * Experiments limited to image classification on ImageNet and speech classification on SpeechCommands using a small network.\n* Practicality of proposed ByteTransformer:\n  * Sequence length and number of flops greatly exceed that of typical modality specific transformer baselines.\n  * Still need to decode input formats in every iteration to apply data augmentations during training, which are crucial for obtaining good performance.\n  * Network architecture is now sensitive to input encoding scheme, rather than simply the input modality; e.g., JPEG compression requires smaller convolutions after the byte embedding than TIFF or PNG.\n* Ablations contain confounding factors:\n  * In general, I would actually expect incorporation of windowed attention and pooling to degrade performance over the typical all-to-all transformer architecture, but these architecture changes to not allow for clear conclusions about the effect of byte encodings compared to raw RGB or time-series/spectral inputs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores whether Transformer networks can be trained directly on byte files without having to decode the inputs. The proposed architecture (ByteFormer) contains windowed attention, strided convolutions, and pooling layers, to process long sequences of bytes. Numerical results conducted on Image and Speech Classification (IN1K and SCv2) using a model with roughly 5M to 9M parameters depending on the specific ablation examining the size of convolutional layers (equivalent to a ViT-Tiny).\n\nThis paper finds that it is indeed possible to train transformer architectures directly on a 1D sequence of bytes, rather than decoding and preprocessing images and audio files into their respective RGB or time-series/spectral representations before network processing. This paper also shows that it is possible for the ByteFormer to operate directly on bytes encoded with complex compression schemes (e.g., JPEG).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Minor questions and suggestions:\n* Interesting experiments in the appendix on obfuscation (permutation and masking). Clarification about the setting:\n  * Is the permutation randomly chosen in each training iteration, or is it shared? Table 4 in the main paper seems to suggest that random permutations during training break locality and degrade performance.\n  * Is the mask randomly chosen in each training iteration, or is it also shared? Does this mean that you no longer use positional embeddings in these experiments?\n* One obvious question is how would a ViT-Ti with a smaller patch size (longer sequence length) compare to the proposed ByteFormer. I was happy to see results to this end in the appendix, but the patch size only goes to 8. I would be curious to see a ViT-Ti with a patch size of 2 or 4, even if this means including windowed attention and strided convolutions to process the longer sequence lengths. This may also lead to cleaner ablations with the proposed method.\n* It is left as future work, but one of the main motivations for this type of exploration in my opinion is for efficient training. This is especially problematic in learning from video, where decoding large videos can be expensive. Not necessary, but numerical results on video classification (e.g., K400), without needing to go so far as characterize efficiency improvements, would strengthen the work.\n* Broadening the set of image and speech tasks would also strengthen the work. Of course different tasks require different types of features. Classification is quite coarse, but how does learning from bytes impact lower level tasks such as detection? It could be that learning from bytes would actually improve performance on such tasks.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zvbTi4HL0a", "reviewer": "Reviewer_gxNr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n* The paper studies a conceptually simple yet insightful model of how far transformers can learn from sequences. File representations are invertible, so it seems natural that a sequence model can directly learn from them. Showing that performance is competitive for file representations demonstrates that even less inductive bias is necessary for certain tasks than was previously assumed.\n* A model architecture is provided (and studied to an extend) that can handle the sequences in question.\n* The ablations demonstrate some interesting learned phenomenon that is file-type specific. These observations could motivate subsequent work. The results in the appendix on potential privacy implications are also interesting.\n\n**Weaknesses:**\n* The method isn't as universal as it could be given the complexity of handling long sequence lengths as well as the need for augmentations.\n* While the ablations are insightful, it's not clear if the model is utilizing information outside what is assumed for image or speech tasks. For example, file sizes or metadata may contain enough information for classification. Figure 6b is one example where it's not clear how 3% of pixels are sufficient for 64% accuracy.\n* Some of the tables are missing comparison points (e.g., Perceiver throughput in Table 5).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to perform training and inference directly on files using a Vision Transformer (ViT) style architecture. Evaluation is performed over ImageNet as well as Speech Commands V2. Results are competitive to baseline approaches. There are 2 challenges faced with this model: 1) long context lengths  and 2) train-time augmentations. The first is handled with shifted attention and downsampling. The second is handled with modality-specific decoding, augmentation, and encoding. The paper also performs ablations that demonstrate some aspects of learning, such as what byte positions are important.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I don't see any critical issues, but I strongly suggest the following:\n* Fill in the omitted results in the tables. This would help with understanding throughput, for example.\n* Additional ablations on where accuracy is coming from. I expect an encrypted file, for example, to have little predictive power. It could be clearer how the model is working if structural elements of the file were ablated (e.g., headers).\n* For related work, it seems appropriate to compare this work to binary analysis methods (e.g., malware).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hVnluOWplG", "reviewer": "Reviewer_D51o", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The critique of the paper \"Bytes Are All You Need: Transformers Operating Directly On File Bytes\" raises substantial points regarding the perceived novelty and depth of the proposed method, its evaluation, and the fundamental premise of considering file bytes as a modality-independent representation. Here is an expanded discussion of the outlined weaknesses:\n\n### Weakness 1: Misinterpretation of Modality-Independent Representation\n- The authors propose that learning directly from file bytes circumvents the need for modality-specific preprocessing, presenting this as a form of modality-independent representation learning. However, this perspective might be somewhat misleading. In the realm of natural images, RGB data could be considered closer to the \"raw\" data, directly corresponding to the visual spectrum captured by sensors. From this viewpoint, file bytes represent a further abstraction, designed to compress and store this visual information efficiently within the constraints of digital hardware. Arguing that learning from such an abstraction is more fundamentally aligned with the data's nature than learning from RGB data challenges conventional perceptions. The transformation of RGB data into file bytes is, in essence, a form of post-processing tailored to suit storage and transmission requirements, rather than an effort to capture a purer form of the underlying information.\n\n### Weakness 2: Lack of Novelty in Model Architecture and Training Strategy\n- A deeper analysis reveals that the paper's proposed method, while innovative in its approach to input data, does not offer significant advancements in the underlying model architecture or training methodologies. The ByteFormer, at its core, relies on established Transformer architecture principles, with the primary novelty lying in its application to a new form of input. This application, though interesting, may not suffice as a groundbreaking contribution without corresponding innovations in how the model architecture or training processes are specifically adapted or optimized for this unique form of input. The paper could have benefited from a more thorough exploration of how traditional Transformer components might be reimagined or reconfigured to more effectively learn from file bytes, or how training strategies could be tailored to leverage the unique characteristics of this data representation.\n\n### Weakness 3: Limited Scope of Experimental Evaluation\n- The experimental validation of the ByteFormer model raises concerns regarding its breadth and depth:\n    1. **Limited Diversity in Benchmarks:** The evaluation focuses narrowly on a single dataset for vision (ImageNet) and one for speech (Speech Commands V2), which may not sufficiently demonstrate the model's versatility across different types of data or tasks. This limited scope makes it challenging to gauge the model's general applicability or its performance in scenarios that diverge from the characteristics of these two datasets.\n    2. **Absence of Ablation Studies:** The paper notably lacks ablation studies that could offer insights into which components of the proposed approach are most critical to its success. Such studies are essential for understanding the impact of different design choices, including the use of file bytes as input, the architecture modifications to accommodate these inputs, and any training strategy adjustments. Without these studies, it is difficult to discern the relative contributions of the method's components to its overall performance or to identify potential areas for optimization and improvement.\n\nIn summary, while \"Bytes Are All You Need\" presents an intriguing concept, its contributions are somewhat undermined by a narrow interpretation of modality-independence, a lack of architectural and methodological novelty, and a constrained experimental evaluation. Addressing these weaknesses could greatly enhance the paper's significance and the perceived value of learning directly from file bytes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper \"Bytes Are All You Need: Transformers Operating Directly On File Bytes\" explores a novel approach to modality-independent representation learning by directly classifying file bytes without the need for modality-specific preprocessing or decoding files at inference time. This method, centered around the proposed model ByteFormer, demonstrates an ability to improve classification accuracy across various data modalities, including images and audio, without any hand-designed processing. The key contributions include the demonstration of ByteFormer's superior performance on ImageNet classification, achieving significant accuracy improvements over established models like DeIT and Perceiver IO with substantially fewer parameters. Additionally, the model's versatility is highlighted through its applicability to audio classification on the Speech Commands V2 dataset, matching or nearing state-of-the-art accuracies without architecture or hyperparameter modifications. The paper also delves into the analysis of learned embeddings and the impact of byte ordering on the model's accuracy, offering insights into the underlying mechanisms of its success. Finally, the researchers commit to open-sourcing their code, facilitating further exploration and application of their findings within the community.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors should more clearly justify why learning on file bytes is scientifically important, beyond possible better performance.", "broader_impact_concerns": "Given the lack of novelty and questionable motivation, I don't think this paper could have significant impact to the machine learning or computer vision community.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers affirm the criteria of claims-evidence agreement and audience. uWSf and gxNr recommend acceptance accordingly but D51o leans toward rejection. In line with the official TMLR criteria and the review content the action editor recognizes but dismisses the vote for rejection.

Modeling bytes moves representation learning in the direction of ever greater generality w.r.t to the input. This work addresses image and audio classification by examining multiple input representations, comparison methods, and analyses including ablations. Furthermore a multi-modal setup across images and audio with bytes processing is explored. Although there are of course more modalities for input, and more and richer tasks than classification, this work advances the field by reducing modality-specificity vs. existing work. This work proposes a transformer, while prior work likewise studied transformers for modality-agnostic processing, but it takes the further step of doing away with modality-specific processing like decoding and reshaping as done by the Perceiver/Perceiver IO. All-in-all this work is informative to machine learning, vision, and audio researchers interested in (1) modality-agnostic and multi-modal modeling and (2) deep learning on compressed/storage-efficient inputs and its publication by TMLR can serve to drive more work to improve accuracy and expand the set of input modalities and output tasks handled by modality-agnostic modeling.

The action editor thanks the authors, for their submission and thorough response, and thanks the reviewers, for their service to TMLR and the detailed reviews and feedback. In particular the action editor emphasizes the quality of reviews in providing specific and grounded points, even if there is ultimately disagreement as in the case of D51o with regard to the need for modality independence. The decision is not a verdict on any particular position, but a synthesis of the submission itself, the reviews, and the response. Lastly the action editor commends the author responses for their level of detail and their pointing to specific content in the paper: these comments can inform future readers.

The action editor votes for acceptance and requests only minor revisions:

1. Please flag the use of modality-specific training and augmentation. This point is discussed clearly in Sec. 4.3, and the abstract does qualify claims with "inference time", but it would be clearer and more accessible to potential readers if this point were spelled out in the abstract or introduction. In the same vein this could be further discussed in the limitations section. Even so, the proposed method is still more modality-agnostic than prior work, and as such this point is raised for clarity and not as a weakness.
1. Please include binary analysis in the related work, as recommended by gxNr, and as promised by the [author comment](https://openreview.net/forum?id=RkaqxxAOfN&noteId=mxPv7IWt2W).
1. Please consider identifying specific additional benchmarks for modality-agnostic modeling in tasks or in datasets. The limitations section mentions these next steps generically but not specifically. For instance the audio experiments do not include AudioSet, which was evaluated by prior work on more modality-agnostic processing (Perceiver: Jaegle et al. ICML 2021). This is entirely optional but could better direct future work.
1. Please consider the perspective of reviewer D51o on modality-specific vs. modality-agnostic representation. This may be indicative of other potential readers' perspective, and as such it may deserve comment in the introduction or conclusion. This is entirely optional but could improve the positioning of the paper for a broader audience.
Decision: Accept (poster)