Neural Video Encoding
Authors: Abel Brown
Robert DiPietro
Venue:  (2020)
Keywords: Kolmogorov complexity
differentiable programming
convolutional neural networks

Abstract: Deep neural networks have had unprecedented success in computer vision, natural language processing, and speech largely due to the ability to search for suitable task algorithms via differentiable programming. In this paper, we borrow ideas from Kolmogorov complexity theory and normalizing flows to explore the possibilities of finding arbitrary algorithms that represent data. In particular, algorithms which encode sequences of video image frames. Ultimately, we demonstrate neural video encoded using convolutional neural networks to transform autoregressive noise processes and show that this method has surprising cryptographic analogs for information security.

Under review as a conference paper at ICLR 2020
NEURAL VIDEO ENCODING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks have had unprecedented success in computer vision, natural language processing. and speech largely due to the ability search for suitable task algorithms via differentiable programming. In this paper, we borrow ideas from Kolmogorov complexity theory and normalizing flows to explore the possi- bilities of finding arbitrary algorithms that represent data. In particular, algorithms which encode sequences of video image frames. Ultimately, we demonstrate neu- ral video encoded using convolutional neural networks transform autoregressive noise processes and show that this method has surprising cryptographic analogs for information security.
In algorithmic information theory, the Kolmogorov complexity of an object the length of the short- est computer program that, without additional input, produces the object as output (Ming & Paul 2008]. For example, a long sequence of ones contains very little information because a short pro- gram for i + 0, do print
can output the data. Similarly, the transcendental number = 3.1415... is an infinite sequence of seemingly random decimal digits but contains only small amount of information since there exists very short program that produces the consecutive digits of π forever (Bailey et al.) 1997). While this idea of exchanging data with computation is theoretically intriguing, from practical point of view, explicit specification of instructions to obtain some target representation is generally an intractable approach for arbitrary data.
Recently, however, artificial intelligence had broad success in computer vision, natural language processing, and speech largely due to the ability to search for suitable task algorithms using differ- entiable programming together with large datasets to obtain abstract computations in the form of neural networks. In this paper, we turn the process around and leverage this same differentiable pro- gramming approach to find abstract computations which represent data. That is, we seek algorithms which encode arbitrary information for storage and retrieval. In particular, we demonstrate methods for the encoding of sequential video data within the weights of convolutional neural networks.
In the simplest formulation, video data, X, is treated as an finite ordered sequence of image frames
X = {To, TI,...,INI
(1)
and we seek an autoregressive computation of the form
Vi+1 fo(y:)
(2)
where yo < and f is some convolutional neural network parametrized by weights 0 such that Vi R x;
possible that f can be trained using a self-supervised learning approach where consecutive image frames form training examples and the sum-of-squares, ||y; -1;==;;1²² is minimized. That is, during training, if learns a mapping from I; to Ti+1, or rather, learns to advance the current frame by one step. The idea is somewhat similar to conventional autoencoders (Hinton & Zemel 1993), but there, f is trained with the identity mapping, from I; on to F;. The catch here is that after training, during recall, f does not receive z; input but instead yı from the previous evaluation. Since ys only approximates Vi, reconstruction error accumulates at each iteration of eq. 2 until sequence reproduction fails]]
The natural inclination at this point is to extend the approach to train over the full sequence by evaluating eq. N times to produce (yI,... yNd and minimizing the mean squared error over the
TThis problem is associated with teacher forcing. See, e.g., Goodfellow al. 2016).

---

Under review as a conference paper at ICLR 2020
entire sequence at each training iteration. This way, how the network is trained and how the sequence is recalled post-trane training are congruent. Unfortunately, for sequences beyond few elements, this is not tractable as long unrolled gradients readily vanish or explode and there is no opportunity for batching which significantly limits training performance and scalability.
The situation whereby the training sequence length is slowly increased after achieving some nominal loss objec- tive over shorter intervals. It important to note that, because CNNs have no recurrent hidden state, this approach can be successfully batched during training over random subsequences of X. There- fore, modulating the training subsequence length to one yields the aforementioned self-supervised learning approach whereas training sequence length equal to the number of frame elements yields the full sequence loss. While this batched curriculum learning strategy does mitigate challenges associated with teacher forcing, gradients can still be temperamental for larger training sequence lengths necessitating the use of gradient clipping and learning rate scaling (You et al., 2017 2019).
be salvaged by adopting curriculum learning strategy (Bengio al., 2009)
An interesting situation occurs when T; z. but f(z; A f(rz), That is, there exists non- unique mapping or sequence ambiguity which is not resolvable from a purely supervised learning point of view since a unique input evaluates to multiple output. It is perhaps surprising to some readers that eq. above is capable of handling this situation successfully, despite CNNs having no recurrent hidden state. Due to the lack of teacher forcing and high dimensionality of the data, the network learns to encode (and decode) state information within the pixels thus willfully incurring a local reconstruction penalty in order to facilitate sequence completion. The success of the batched sequence learning strategy depends on the nature of the ambiguity.
Just the state of dynamical system is not well defined by a single observation, often encoding is facilitated by sequence dilation of the learned mapping which more uniquely defines the trajectory of sequence evolution. A single frame mapping from 2, to I++1 exhibits a dilation of d 1 and can be naturally extended to d = 2 by channel concatenation to learn a computation of the form [yx: y/+1] = f(y 3d) with dimension (2C') x И x W; and so on ? This helps individuate samples and thereby allay the need for contextual sequence information to achieve recall. For example, the alternating sequence ambiguity in { A, B, A, C, A, D) is easily resolved by dilation d = 2 since all training examples, f (ABB) + BA, f (BA) + AC' and so on, are unique.
The KTH (Schüldt et al., 2004) and DAVIS17 (Pont-Tuset et al., 2017) datasets are used to inves- tigate the methods described above. KTH data provide a simple initial investigative platform with monochromatic vidcos featuring simple repetitious actions performed by humans (clapping, hand waving, punching. etc) against static background while the DAVIS17 data features multi-channel RGB capture of generic action scenes across a variety of dynamic environments (surfing, breakdanc- ing, paragliding, etc). See Figure 1 and Figure 2 for encoding examples using a simple autoencoder style network defined in (I) and applied according to eq. 2 Although compression and fidelity are peripheral considerations Wu et al. (2018) for full treatment), the examples presented generally yield faithful reproductions according to multi-scale similarly measure (Wang et al.) (003) and achieve respectable compression ratios of 10x- 15x simply by modulating the number of convolu- tional layers in the CNN. Compression ratios can be further improved by applying mixed precision training techniques (Micikevicius et al. 2017).
A variety of practical engineering considerations arise during implementation. For example, it is possible to analyze sequence loss during post-production and inject a ground truth frame at par- ticular deterioration threshold to renew the recall process; analogous to the use of key-frames in standard compression algorithms. It is also possible to train additional networks which upscale the base reproduction. This upscaling approach has advantages in controlling network parameter growth (and hence the compression ratio) as two convolution layers each with 32 filters have only half the parameters compared to a single layer of 64 filters. In this of two functions
way, eq. is written the composition
Vi+1 fe(
31" = fe. (v.)
where So advances the sequence and the subsequent application of Sa, bolsters fidelity. From this point of view. fe. performs the role of denoising autoencoder (Vincent al. 2008) which strives to
In some ways, dilation akin to the order of autoregressive model
2

---

Under review as a conference paper at ICLR 2020
(A)
frame =1, mssim =0.995
frame =99, mssim =0.995
frame = 98, mssim =0.995
B frame =16, mssim =0.77
frame =96, mssim =0.22
frame =165, mssim =0.77
(C) frame =1, mssim =0.76
frame =98, mssim =0.76
frame =196, mssim =0.76
Figure 1: Video encoding reproductions 128x128 pixel resolution from the KTH dataset. (A) A reproduction of p handclapping d4 using 128 convolutional filters per layer which, although not parameter efficient, achieve high fidelity after 99800 training epochs full FP32 precision. (B) A reproduction of p handwaving d4 using only 32 convolutional filters per layer and dilation after 363947 training epochs with mixed FP16 precision. This encoding achieves a compres- sion ratio of 27x but at the cost of some fidelity. (C) Finally, reproduction of person25 Joxxing _dl using 32 convolution filters per layer and dilation precision.
after 473701 training epochs with FP16
3

---

Under review as a conference paper at ICLR 2020
(A)
frame =1, mssim =0.99
frame =43, mssim =0.99
frame 86, mssim =0.98
(B)
frame =1, mssim =0.97
frame =33, mssim =0.90
frame =66, mssim =0.37
(C)
frame =0, mssim =0.98
frame =17, mssim =0.98
frame =33, mssim =0.98
ment
meu
(D)
frame =1, mssim =0.94
frame =25, mssim =0.44
frame =65, mssim =0,55
Figure 2: Video encoding reproductions from the DAVIS' 17 dataset. (A) Rhino scene reproduction at 3x 128 pixel resolution with 128 convolutional filters per layer and 200000 training epochs. (B) Miami surf scene recalled 3x128x128 pixel resolution using 128 filters per layer and 77000 training epochs. (C) Recalled frames at 3x 12x 128 pixel resolution from the Rollerblade scene using 128 convolutional filters per layer and 250000 training steps. (D) Breakdance reproduction at 120p with pixel resolution 3x 120x213 using 48 convolutional filters per layer at 85000 training epochs.
4

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Pseudo definition of the autoencoder style network used throughout this work and applied as defined in eq. (2). All layers define neoov convolutional filter channels of size 3 x 3. The first and final layers takes C X d number of imput and output channels respectively where C is the number of image channels and d the sequence dilation. Where necessary, stride and padding used to maintain dimensional equality of input and output tensors.
function NETWORK(ncore)
relu(Conv22(x)))
X = rclu(Conv2d(x))
relu(Conv2d(x))
upsample
relu(Conn2d(x))
X (elu[[onn/Transsoeeed()))
upsample
X relu(Con/2d(x))
X Conv Transpose2((x)
return X
(A) frame =35, mssim =0.92
frame =52, mssim =0.87
frame 866, mssim =0.84
(B)
frame =35, mssim =0.96
frame =52, mssim =0.94
frame 886 mssim =0.93
(C)
frame =35
frame =52
frame 866
Figure 3: Video encoding reproduction of the 1AVIS'17 Parkour scene 120p using the upscaling approach described by eq. eq.3 with M = 1. (A) Frames, produced by the base network fo, with 32 convolution filters per layer, which advances the sequence each application. (B) Upscaled frames, y/ produced by denosing autoencoder, fe., with 32 convolution filters per layer. (C) Associated ground truth frames, Z4.
form 'repaired' output from 'corrupted' input. This allows computational demands during recall to be modulated on a Frame-by-framm basis according to platform limitations, real-time constraints, and This application of upscaling is shown in Figure
From here, we can generalize further and leverage the recent advancements in normalizing flows whereby simple initial density is transformed into more complex by applying sequence of transformations until a desired level of complexity attained (Tabak & Turner, 2013 Dinh et al. 2014 Jimenez Rezende & Mohamed 2015: Kingma et al. 2016). The density q obtained by
5

---

Under review as a conference paper at ICLR 2020
successively transforming random variable 20 with distribution gd through chain of K transfor- mations fk is:
(4) where eq. (4) is shorthand for the composition fM (MM- f1 In this way, the initial den- sity is said to 'flow' through the sequence of mappings. For our purposes, the original sequence X defined in ④ can be seen as just an arbitrary random noise process in some high-dimensional space. Rather than model the process X directly, instead, it can be approximated, to some acceptable level of fidelity, through a chain of transformations over some known latent process, = { 2N). That is, given random process, Z, defined by g. we have
zx o...... o fi
2,+1 g(2)) z 11 c... f2 2 f1(z))
where 2f 2, represents j consecutive transformations applied to 2, and where each fj is deep neural network defined by Algorithm]] [ These transformations are simply a series of bijective mappings (under the given restricted domain and range) and therefore alleviate the aforementioned sequence training challenges associated with realizing computations of the form defined by eq. 2
The question then becomes, what g?
The choice of g must be some random but deterministic high dimensional process with the require- ment that Vzz 2j Z, 2; * zz. That is, the elements of Z unique such that the process is noncyclic in order to support a bijective mapping. A key insight is that given some g defined by Al- gorithm and applied according to eq. (5), the process Z is completely defined by the starting point, 2o. Therefore, from the point of view of Kolmogorov complexity, { (20.g} g} is a random computation for the production of the sequence, Z, and normalizing flows with differential programming provide
the framework transform this computation: X L {=8.g): From compiler point of view, z is simply an abstract intermediate representation (IR).
A simple recipe for realizing g is choose any two random points 20, 2' RCxHxW and learn parameters, o, such that N recursive applications of Is starting with 20 yields 2'. Although, in practice, after sufficient number of training iterations of 9o- it does not matter if 2N computed according to eq. (5) does not equal z' rather it is only required that Os satisfy the noncyelic random walk properties described above. This recipe for g is described in Algorithm 2 and demonstrated in Figure 4 The full workflow of transforming random process, Z, generated by g into the target sequence X according to eq. (5) is shown in Figure
Algorithm Simple recipe for realizing a high-dimensional autoregressive noise process.
function FORWARD(g.2.n) for i < 0, n do
D defined according to eq. (5)
g(2)
return z
N length( X)
20 uniform(C H W)
> samples from uniform distribution over the interval [0, 1)
uniform(C H x W)
9+ NETWORK(ncone)
described in Algorithm
>> determine that minimizes the || 2/112
for i + 0. FORWARD(Gs:: 20, N)
1 < ||z =
8 OL
+ Step(á,6)
CRYPTOGRAPHIC CONSIDERATIONS
The neural video encoding process shares many analogs with symmetric cryptographic algorithms. In general, cryptography the body of techniques for secure communications focused the

---

Under review as a conference paper at ICLR 2020
epoch 2000
epoch 6000
epoch = 8000
epoch 10000
15
10
epoch 12000
epoch 17000
Figure 4: Visualizing the training of high-dimensional autoregressive noise process. The compu- tation, as defined is simple convolutional neural network according to Algorithm [ and trained defined by Algorithm - In this instance, Io was initialized with nconv the learning process, distance matrices are calculated over the elements of Z at various stages of training. After 17000 epochs of training, the distance matrix confirms that the elements generated by I& are sufficient to support bijective transformation.
To help visualize
struction of protocols that prevent third parties or the public from reading private information. A protocol defines system of information exchange which includes rules, syntax, semantics and syn- chronization of communication. Moreover, a cryptographic protocol usually focuses on various aspects of information security such as confidentiality. data integrity and authentication. Cryptogra- phy is often synonymous with encryption which is the conversion or encoding of information from coherent readable state to incoherent state of apparent randomness and the reverse process of decryption. A cipher is defined by a pair of algorithms specifying an encryption and decryption process controlled by the of auxiliary private key which is required to decrypt the associated data. A key must be selected before using a cipher to encrypt data and without the key, it should be extremely difficult impossible decrypt the contents. When the same key is used for both
7

---

Under review as a conference paper at ICLR 2020
(A)
frame 30
frame = 62
frame 81
(B)
mssim 0.88
mssim 0.89
mssim 0.89
(C)
mssim 0.94
mssim 0.94
mssim 0.94
(D)
Figure 5: Neural video encoding via normalizing flows of the DAVIS' 17 Breakdance scene at 120p. (A) Visualizations of the variables {2g 20 281 from some random autoregressive noise process,
Io. having nconv = and procured according to Algorithm having 10000 training epochs. (B) The variables { = = as computed by eq.(5) where 2 fe, (z") Se, o (o(="_)) and fo is configured with nconv = 32 having been trained for 30000 epochs. (C) The variables
{2g0 #62 2 computed as 2/ = (z! = fe, fe, (2.") with fo, again having ncone = 32 and trained for 30000 epochs. (D) Ground truth sequence elements {I30, T62, 181) from the unen- coded data,
encryption and decryption, the cipher defines a symmetric key algorithm! Symmetric cryptography deals with the efficient construction of pseudo random functions which form the building blocks of symmetric algorithms and have the form & lı(z) with the following properties (Ramkumar 2014
1. Given h(2), small changes produce output x' unrelated to
2. Given z, there is no way to make reliable predictions regarding any part of x.
3. Given ø, the easiest way to find that satisfies x h((²)) is brute-force search.
4. The fastest way to determine preimages and x' satisfying h(x) h(x') is brute-force search.
See Paar
Pelzi 2010
Schneier 2015), and Ferguson 2010 for additional discussion.
8

---

Under review as a conference paper at ICLR 2020
From a high-level perspective, it is easy to identify latent variable Z and target sequence X as the encrypted and unencrypted data respectively, while the variable, =f performs the role of shared key. The encryption algorithm is defined by the realization of generator function, 9. which consumes 20 to produce Z, and the decryption algorithm is defined by the application of learned transformations, F F= F applied according eq. (5). Together, the neural video components, {g, F), form a neural cipher and with shared use of key, z8. define a symmetric cryptographic algorithm. An informal analysis indicates the necessary alignment with the properties of pseudo random functions. For particular instance of Io- changes to =6 yield unrelated sequences Z' and, by definition, the random values of Z alone provide information regarding X. Moreover, as black box functions, given some target element Z;, the only way to find z' satisfying I; = F(=') is indeed brute-force search. It is natural go further and consider the significantly more empowered situation in which an adversary could perform differential calculations over F to optimize the search for 2' which minimizes || F()')I. Effectively, this means the decryption algorithm and a portion of the un- encrypted data have been compromised. Even so, without the associated generator, g. it remains very difficult or impossible to resolve additional content since despite having obtained 2', it is not possible to make reliable predictions regarding any part of z. Finally, an example of these properties provided in figure/ddemonstraiie key. In this situation, both the encryption and decryption algorithms have been compromised but the adversary has no knowledge of or X. A robust cryptographic algorithm must remain secure even if everything about the implementation, except the key, becomes available to an adversary Indeed, some work has suggested that CNNs retain accurate image information with different degrees of geometric and photometric invariance Mahhndrran & Vedaldi 2014). However, the authors attempts at extracting meaningful content using total variation based approaches were unsuccessful since having the encryption and decryption algorithms alone provides little basis for the construction of a loss function to minimize (see also Ulyanov et al.] (2017)). This discussion is, of course, not in- tended to provide thorough cryptographic analysis of neural video encoding but rather to facilitate understanding of various properties and highlight the potential for information security.
a failed attempt to decrypt an encoded video with an imposter
RELATED WORK
The storage of information in neural networks has rich history (Kohonen & Lehtiö 1989). Early parallel models of associative memory were used in the investigation of long-term memory systems in cognitive science probing questions of how information is represented in the brain and what kinds of processes operate on it (Anderson & Hinton 1989). Some of this early work was motivated by holography as a mechanism for distributed associative recall (Willshaw) 1989) made possible by Nobel prize wining advancements in the development of the holographic method by Gabor. Inter- estingly, as highlighted by Hinton| (1989) in the same body of work, Marr, Palm, and Poggio (Marr) et al. 1978) were particularly interested in equations of the form C = o ((L()")) where L is a linear operator and a a nonlinear function and stressed the importance of understanding their be- havior. These associative memory models continue to garner some attention (Karbasi et al. 2013
Mazumdar Rawat, 2015)
Next frame prediction in moving imagery and video data shares similar motivations with neural video encoding but differ in ultimate objective. Lotter et (2017) for discussion and relevant references. Although observed but not highlighted, video encoding networks trained on KTH data for simple repetitious actions often exhibited the ability to generalize beyond encoded sequence for short periods and demonstrating limited capacity for what could be argued as generalization versus pure recall.
The application of neural networks to cryptographic problems is not new (Kanter al. 2002 Klimov et al., 2002) Abadi & Andersen] (016) and the synergies between machine learning and cryptography have been appreciated for some time (Rivest) 1993). Many applications of machine learning to cryptography are realized through cryptanalysis tools and majority of neural network applications have been orchestrated at the bit level and around anti-symmetric public protocols as in (Kanter et al.] (002). Neural video encoding defines a simple symmetric protocol operating in situ at the pixel and hence leverages the native advantages of distributed representation. Although,
*This is known as Kerekhoffs's principle and (Claude) Shannon's 's maxim

---

Under review as a conference paper at ICLR 2020
(A)
z'
(B)
Figure 6: An attempt decrypt (i.e. recall) an encoded video using imposter key. (A) Key =8 and imposter key =Ø6 (B) On the left, =8 and Io applied to generate the appropriate keys {= =82 #$1) } and decrypted through application
F. On the right, the imposter key, =/0 with the same Jø yields -0 z0 which decrypt to incoherent states under F.
an invalid key stream {=30-
#62 281)
visual cryptographic methods applied images directly have existed for some time as well (Naor & Shamir 1995) (see also Punithavathi & Subbiah (2017) for recent survey).
CONCLUSION
This work demonstrates encoding methods for the storage and retrieval video data with convolu- tional neural networks. There remain many open considerations but ultimately the ability to realize computations of arbitrary complexity which represent data is clear. Moreover, these computations are abstract representations of data and therefore exhibit desirable synergies with cryptographic systems underpinning confidentiality to facilitate information security. Recent advancements in all- optical neural networks (Lin et al. 2018 Zuo et al., 2019) motivate the future exploration of com- bining past holographic based associative memories efforts with the encoding methods presented here to create potential new forms of optical memory and beyond.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martín Abadi and David G. Andersen. Learning to Protect Communications with Adversarial Neural Cryptography. arXiv e-prints, art. arXiv:1610.06918 Oct 2016.
James Anderson and Geoffrey Hinton. Parallel Models of Associative Memory, chapter Models of Information Processing in the Brain, pp. 23-62. L. Erlbaum Associates Inc., 1989.
David Bailey, Peter Borwein, and Simon Plouffe. On the rapid computation various polylogarith- mic constants. Mathematics of Computation, 66:903-9990 9113,9913111111 1997.
Yoshua Bengio, Jerome Louradour, Ronan Collobert, and Weston Jason. Curriculum learning. In Proceedings of the 26th International Conference on Machine Learning, 2009.
2009.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear Independent Components Estimation. arXiv e-prints, art. arXiv:1410.8511 441..851... Oct 2014.
Niels Ferguson, Bruce Schneier, and Tadayoshi Kohno. Cryptography Engineering. John Wiley and Sons, Inc, 2010.
lan Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Geoffrey Hinton and Richard Zemel. Autoencoders, minimum description length and helmholtz free energy. In Proceedings of the 6th International Conference on Neural Information Processing Systems, 1993.
Geoffrey E. Hinton. Parallel Models of Associative Memory, L. Erlbaum Associates Inc., Hillsdale, NJ, USA, 1989. ISBN 0805802703.
Danilo Jimenez Rezende and Shakir Mohamed. Variational Inference with Normalizing Flows. arXiv e-prints, art. MeXXv:150550577 May 2015.
Ido Kanter, Wolfgang Kinzel, and Eran Kanter. Secure exchange of information by synchronization of neural networks. EPL (Europhysics Letters), 57, 02 2002. doi: 10. 00112099cp//2202--055559999
Amin Karbasi, Amir Hesam Salavati, and Amin Shokrollahi. Iterative learning and denoising in con- volutional neural associative memories. In Sanjoy Dasgupta and David McAllester (eds.), Pro- ceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 445 453, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. m.r. wwwpppresssvve. http/////kaaaaal3..3 html
URL http://proceedingge
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Chen, Ilya Sutskever, and Max Welling, Improved variational inference with inverse autoregressive flow. In D. D. Lee, M. Sugiyama, U. V. Luxburg, Guyon, and R. Garnett (eds.), Advances in Curran Associates, Inc.,
Neural Information Processing Systems 29, pp-
2016. URL //tp:r////pppppepppep pemppers.iiss. wwtpsppcppppppe///ppppppppppppep inference-witee th- wwwhhhiivereeeuuueeeeeee autorrgge-ttoooggresss.e ve fle.flo..ppd improved-variationnl
Alexander Klimov, Anton Mityagin, and Adi Shamir. Analysis of neural cryptography. In Yuliang Zheng (ed.), Advances in Cryptology - ASIACRYPT 2002, pp. 288-298, Berlin, Heidelherg, 2002. Springer Berlin Heidelberg, ISBN 978-3-540-361778.
Teuvo Kohonen and Pekka Lehtiö. Parallel Models of Associative Memory, chapter Storage and Processing of Information in Distributed Associative Memory Systems, pp. 129-167. L. Erlbaum Associates Inc., 1989.
Xing Lin, Yair Rivenson, Nezih Yardimci, Muhammed Veli. Yi Luo, Mona Jarrahi, and Aydogan Ozcan. All-optical machine learning using diffractive deep neural networks. Science, (6466)))) 1004-1008, 2018. ISSN 0036-8075. 8075. doi: 10. 10.11120 http:///ccccccaat0000... URL /tllsttts///sceeeee
sci /

---

Under review as a conference paper at ICLR 2020
William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video pre- diction and unsupervised learning. In 5th International Conference on Learning Representa- tions, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017. URL http::///ppeeevieeeeee net h forum? 1d=Bl ewdtt 9xe
Aravindh Mahendran and Andrea Vedaldi. Understanding Deep Image Representations by Inverting Them. arXiv e-prints, art. arXiv: 1412.003, Nov 2014.
D Marr, G. Palm, and Tomaso Poggio. Analysis of cooperative stereo algorithm. Biological cybernetics, 28:223-39, 04 1978. doi: 10. N0.10077BFF0334426.
Arya Mazumdar and Ankit Singh Rawat. Associative memory via a sparse recovery model. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Gareet (eds.), Ad- vances in Neural Information Processing Systems 28, pp. 2701-2709. Curran Associates, Inc., 2015. URL spwrse- 1 pdd
n ps. wttpppppe//1.55.5. a me-
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksäi Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed Precision Training. arXiv e-prints, art. 1r00v:1700003740, Oct 2017.
Li Ming and Vitányi Paul. An Introduction to Kolmogorov Complexity and Its Applications. Poringer-Verlaas New York, edition, 2008
Moni Naor and Adi Shamir. Visual cryptography. In Alfredo De Santis (ed.), Advances in Cryptol- ogy EUROCRYPT pp. 1-12, Berlin, Heidelberg, 1995. Springer Berlin Heidelberg. ISBN 978-3-540-44717-7.
Christof Paar and Jan Pelzl. Understanding Cryptography. Springer, 2010.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Sorkine- Hornung, and Lue Van Gool. The 2017 davis challenge on video object segmentation. arXiv:1704.0000 2017.
Punithavathi and Geetha Subbiah. Visual cryptography: A brief survey. Information Security Journal: A Global Perspective, 26:305-317, 11 2017. doi: 02.1080/19393355.2117..28
Mahalingam Ramkumar. Symmetric Cryptographic Protocols. Springer, 2014.
Ronald L. Rivest. Cryptography and machine learning. Hideki Imai, Ronald L. Rivest, and Tsutomu Matsumoto (eds.), Advances in Cryptology - ASIACRYPT '91, pp. 427-439, Berlin, Heidelberg, 1993. Springer Berlin Heidelberg. ISBN 978-3-540-48066-2.
Bruce Schneier. Applied Cryptography. John Wiley and Sons, Inc, 2015.
C. Schüldt, Laptev, and B. Caputo. Recognizing human actions: local SVM approach. In Proc. Int. Conf. Pattern Recognition (ICPR 044, Cambridge, U.K, 2004.
E. G. Tabak and Cristina V. Turner. A family of nonparamettii density estimation algorithms. Com- munications on Pure and Applied Mathematics, 66(2):145-164 2013. doi: 10. 001100/cpa.21423
URL htttps:///n/ wtt.///nnnnnnnneoooooooooeenelen ine iirrirrrrr..r wi wer.eyy..oo com htt///o//aabaa/ 100. 1002 /tt.:///pppp... 21423
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep Image Prior. arXiv e-prints, art. arXiv:1711.1092: Nov 2017.
Vincent, H. Larochelle, Bengio, and Manzagol. Extracting and composing robust features with denoising autoencoders. pp. 1096-1 1103, 2008.
Z. Wang, E. P. Simoncelli, and A. C. Bovik. Multiscale structural similarity for image quality assessment. In The Thrity-Sevent Asilomar Conference on Signals, Systems Computers, 2003, volume 2. pp. 1398-1402 Vol.2, Nov 2003. doi: 10.1099/ACSSC2200..2222
David Willshaw. Parallel Models of Associative Memory, chapter Holography, Associative Meme- ory, and Inductive Generalization, pp. 103-124. L. Erlbaum Associates Inc., 1989.
12

---

Under review as a conference paper at ICLR 2020
Chao- Chao-Yuan Wu, Nayan Singhal, and Philipp Krähenbühhl. Video Compression through Image Inter- polation. arXiv e-prints, art. arXiv: 180.01890.0699, Apr 2018.
Yang You, Igor Gitman, and Boris Ginsburg. Large Batch Training of Convolutional Networks. Aug 2017.
arXiv e-prints, art. arXiv:1708..000
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, and Cho-Jui Hsieh. Large Batch Optimization for Deep Learning: Training BERT in 76 minutes. arXiv e-prints, art. 1rXiv:190400096,, Apr 2019.
Ying Zuo, Bohan Li. Yujun Zhao, Yue Jiang, You-Chiuan Chen, Peng Chen, Gyu-Boong Jo, Junwei Liu, and Shengwang Du. All-optical neural network with nonlinear activation func- tions. Optica, 6())1132- Sep 2019. doi: 10.13640PTICA.600112.. URL http:
// /ww.osapubllshingg http://a.a..aaa.ooo..ooooooaoaa. ottp://atcaatttttt.ooooaaaoooaaa 6---119-11116--1222--------
13


Reviews:
[{"review_id": "ryg6ZC3jYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I'm not sure what the contribution of this paper is. It seems to contain a variety of weakly related motivational examples.  The paper begins by stating the computer vision advanced due to \"ability to search for suitable task algorithms via differentiable programming.\" But I don't think this claim is reasonable, i.e., I don't think CNNs are searching for task algorithms. \n\nThen the paper explains Kolmogorov complexity, but this is completely unrelated to the rest of the paper. At no point in this paper is this concept used and this motivation is not useful.\n\nThe paper then introduces the method, which is a standard, simple autoencoder. Some image evaluations of this are shown, but no contribution in terms of the model is made.\n\nFinally, the paper briefly mentions some connections to cryptography, but it is unclear what or why this connection matters.\n\nThe paper has no real experimental analysis, it doesn't seem to propose anything new or make any clear contribution, and overall, is quite unclear on what it is trying to do. The current paper is unsuitable for publication. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SJl7REkMcr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper introduces convolutional neural networks to encode video sequences. In particular, it exploits an autoencoder style convolutional network to compress videos with certain ratios by modulating the number of convolution layers. Furthermore, it studies an autoregressive model to learn the sequential information over videos.\n\nStrength:\nThe idea of using convolutional networks to encode the videos is interesting. \nThe used autoencoder and autoregressive techniques are promising for video encoding. \n\n\nWeakness:\nThe paper gives me an impression that there are very few works to apply convolutional networks (including autoencoder with autoregressive processing) to encode videos. I cannot see any existing works sharing the same motivation, and the paper does not evaluate any related works for comparison. But it seems the use of autoencoder and autoregressive techniques look very straightforward to me. Please clarify the concern. \n\nAbout technical contribution, it is not clear to see if there are any improvement over the conventional autoencoder and autoregressive models. It would be better to make some necessary discussions in the paper. In addition, it seems no descriptions showing how to learn the bijective mappings f_1, ..f_j.\n\nRegarding the evaluation, the paper uses visual results and multi-scale SSIM (mssim) to study the effectiveness of the proposed method on the KTH and DAVIS’17 datasets. While a small number of video frames are used, it would be necessary to evaluate the overall (average) mssim on the whole dataset. \n\nThere are quite a few typos or grammatical problems on sentences such as “...algorithms which encode sequences of video image frames.”, “f is some convolutional neural network…”, “analogous to the use of key-frames in standard compression algorithms. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJgISm7mYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors show that CNNs are somewhat able to compress entire videos within their parameters that can be reconstructed by an autoregressive process. This is an interesting idea that has been explored in a different context before (e.g., Deep Image Prior, Ulyanov et al. 2017). There is also plenty of work in the area of exploiting NNs for video compression/encoding (see [1] for instance). However, a bit unusual is the choice of compressing a video into network parameters, which is quite an expensive process using backprop. I could not find any motivation for why this would be a good idea, potentially because the paper does not state any explicit goal/contribution. In any case the authors show merely some mediocre qualitative results. There are no comparisons to prior work and no empirical results. The combination of methods also seems a bit arbitrary.\n\nTherefore this paper has no valuable contribution in its current form and I vote for rejection.\n\nMajor issues\n- No empirical results\n- No comparison to any baseline or prior work\n- Confusing: the paper has almost no structure, e.g., there are almost no sections at all. Symbols are not used consitently (e.g., functions f and g).\n\nQuestions\n- What's the goal of employing normalizing flows in this paper?\n\n[1] Image and Video Compression with Neural Networks: A Review. Ma et al. 2019.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has several clarity and novelty issues.
Decision: Reject