Kokoyi: Executable LaTeX for End-to-end Deep Learning
Authors: Minjie Wang
Haoming Lu
Yu Gai
Lesheng Jin
Zihao Ye
Zheng Zhang
Venue: ICLR 2022 Submitted (2022)

Abstract: Despite substantial efforts from the deep learning system community to relieve researchers and practitioners from the burden of implementing models with ever-growing complexity, a considerable lingual gap remains between developing models in the language of mathematics and implementing them in the languages of computer. The mission of Kokoyi is to close this gap by enabling automatic translation of mathematics into efficient implementations, thereby making math-in-codes and math-in-model consistent. This paper presents our first step towards the goal: kokoyi-lang, a programming language with the syntax of LaTeX and the semantics of deep learning mathematics, and a prototype kokoyi-lang compiler and runtime supporting advanced optimizations such as auto-batching. Kokoyi is integrated with Jupyter Notebook, and will be released in open-source.

Under review as a conference paper at ICLR 2022
КOKOYI: EXECUTABLE LATEX FOR END-TO-END DEEP LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Despite substantial efforts from the deep learning system community to relieve researchers and practitioners from the burden of implementing models with ever- growing complexity, considerable lingual gap remains between developing mod- els in the language of mathematics and implementing them in the languages of computer. The mission of KOKOYI is close this gap by enabling automatic translation of mathematics into efficient implementations, thereby making math-in- codes and goal: kokoy i 1a-lann -lngg, a programming language with the syntax of 1TEX and the semantics of deep learning mathematics, and prototype kokoyi-lang compiler and runtime supporting advanced optimizations such as auto-batching. KOKOYt is integrated with Jupyter Notebook, and will released open-source.
consistent. This paper presents our first step towards the
INTRODUCTION
The success of deep learning a tale of two ends. At one end is model development, which leverages the language of mathematics to define models. At the other end is model implementation, which relies on programming languages such as Python and CUDA to unleash the power of big data and efficient computation. Despite the success, the language gap between the two ends is arguably the source of many evils, such as slow prototyping, wrong/ineficceen/uneeddaeee implementations, irreproducible results in publications, and learning barriers to name but few. A model is often implemented and re-implemented in different, incompatible deep learning frameworks, an unnecessary fragmentation from model point of view.
The mission of KOKOYl is to close the language gap between the two ends, making math-in-codes and math-in- in-model consistent. KоKOYΙ advocates simple principle: your model is your code. KоKOYι introduces a programming language called kokoy 1itooyi-loe -lang with a programming model elevated to mathematical equations written in 18TEX, rendering language popular among deep learning researchers (Lamport) 1994). Syntactically, kokoyi-lan
is a LTTX subset that is prevalent in mathematical definitions of deep learning models, making kokoyi lang programs renderable just like 1АTE documents. Semantically, kokoyi Miktoiilaas llng formalizes the consensus of the semantics of mathematics among deep learning researchers and practitioners. KOKOYι aims boosting productivity at community level: researchers code model once instead of twice (one in a program and another in the publication), kokoyi - ang is designed be framework agnostic, the model written in cokoyi-lang in paper is executable by all and can be ported to different frameworks.
As teaser, the following (rendered) line of kokoyi-lame ang code defines the mean squared error loss function for linear regression model with weight w E Rd and bias b E IR given a dataset D;
l(w, b. D)
Σ (DN))D
((w + y)²
D
Given kokoyi -lng programs, KOKOYI can generate executables using various backends. In its current prototype, KOKOYI generates PyTorch code because of its widespread adoption. However, compared to Py deep learning frameworks that rely on Python for usability, KOKOYI po- tentially avoid Python's overheads without degrading user experience by translating cokoyi-larn programs into more efficient deep learning IRs such Relay (Roesch et al. 2018) instead of Python.

---

Under review as a conference paper at ICLR 2022
IDE (Jupytor Notebook Environment
[4] Sedrcy
cass: iS
R.4)
Agets [s] iconece aet Length of the Exout Soneeeco)S
(8)
Compller
Kokays
Type
Vidop
Code
luacs Libra; a Lan Vaanis) 13 r
(hap jeteger
(begt//cnne))
Kokoys batchingiigggg
Tonsor Backend (PyToroh)
tend:cases)
On
JPSt (MLL)) Change
Medsle ENNGS My, Mr, 0,6.
inte mubll
[ [5) Get leeth of seqazaco
S (01 Right
(c)
(Disse S)
May the bmnger (Ds deas reproeetationn
1427 ALE Note
prt
Adcaare sertare secuaratian ander. reture SeE NL HCT a.x. self t.
IuutWW AAya W. by is
Au
(b)
Figure 1: (a) A typical work flow when using KOKOYι; (b) Jupyter notebook example of a multi-layer RNN encoder; (c) The corresponding auto-generated module in PyTorch.
The level of abstraction offered by the language of mathematics can enable various code optimizations deemed hard for existing deep learning frameworks. An example is auto- auto-batching, which can relieve users from the burden of manually vectorizing per-sample code, an often counter-intuitive endeavor mandated solely by efficiency. Previous attempts of auto-batching like TensorFlow-Foke (Looks) et al., 2017), DyNet (Neubig et al. (017) and JAX (Bradbury et al., 2018) heavily rely on Just- in-time compilation, which are limited to statically shaped samples. By contrast, KOKOYI takes a holistic design, constructs built during compilation facilitate auto-batching during execution. As a result, KOKOYI can auto-batch complex models, such as the auto-regressive attention module in Transformer (Vaswani et al. 2017).
KOKOYI focuses on model specification. Our informal survey of sampled papers from several machine learning conferences, showing KOKOYI can impact about 15% to 66% of the work where model development is the focus (Appendix AA). As such, we do not attempt to reinvent all the wheels, compiled kokoyi- -llang programs are callable Python objects and integrate with other stages (c.g. data preparation, model tuning, and model serving) of the deep learning pipeline; whenever possible, KOKOYI reuses efficient existing modules.
We implemented Jupyter Notebook frontend to facilitate interactive rendering and execution of kokoyi-lang programs ((see HTML examples in the supplementary materials). There also an interactive Web server where one can prototype model and download both the PyTorch module and the l snippet. We envision supporting compilation of kokoyi ang code embedded in 18TE documents in the future to enable executable research papers.
We implemented in kokoyi- lang several important deep learning models with various complexity, and find most of our per-sample implementations, with PyTorch backend, achieve performance comparable to manually tensorized PyTorch implementations. At the same time, models implemented in kokoyi-1aa -nng are substantially more succinet, for instance the standard Transformer (Vaswani eet al. 2017) needs only 65 lines of code, saving users from the tedious and often error- error-prone process of manually batching computation (Appendix BB. Although large-scale user experience study is yet to be conducted, initial feedbacks from a group of deep learning researchers and data scientists indicate that compared to existing deep learning frameworks, kokeyi - -lang much closer their mathematical formulations of deep learning models. KOKOYI will open source soon.
2
KOKOYI OVERVIEW
KOKOYI aims to be pragmatic, while adhering to the "your model is your code" principle. It is built around and leverages the existing Python ecosystem, yet still reserves the flcxibility to revert to more efficient execution when possible. a depicts the work flow when using KOKOYI. Development starts in an IDE and we currently choose the Jupyter Notebook, a popular interactive environment for live coding and data rendering.
Users can write a KOKOYI program - one or more mathematical equations in a Jupyter Notebook cell, led by the &kokoyi marker to register hook function that renders the equation via MathJax.
2

---

Under review as a conference paper at ICLR 2022
Programming in KOKOYI uses Kokoyi-lang, BTE dialect that is extended with syntax familiar to DL researchers. Running the cell triggers the compilation of the equation to generate a callable python object. In this example (Fig. [T)), the returned object represents the module definition of an T.-layer RNN to compute the representation of sentence S. We hook the program execution with a proper error handler highlight the portion of the equations that causes the error.
KOKOYI translates tensors and functions defined in kokoyi-lang into tensors in underly- ing DL frameworks and Python functions, and makes them accessible in Python dictionary in Jupyter notebooks). For example, the RNN PyTorch module calls [ "RNN" (Fig. [c)) in its forward function. KOKOYI also ports many Py- Torch's 's commonly used modules (e.g. LR)) the porting is mostly straightforward. with the only complexities arise from supporting auto-batching. Our current prototype enables one-click auto-generation of a Pytorch module (Fig. [c)) once it is specified in kokoyi-las
(named
This capacity simplistic for now and the user is required to complete the rest of the initialization (e.g. selecting the initializer). It is relatively straightforward to generate more initialization code in the future.
The KOKOYI compiler generates PyTorch code directly and thus the execution just like any other PyTorch programs. Note that KOKOYI can in principle completely by-pass Python by translating the program into low-level IRs such as Relay IR (Roesch et al. 2018) and LLVM IR (Lattner choose to focus on Python code
& Adve 2004) to enable efficient execution. Nevertheless, generation mainly due to its dominating ecosystem within the deep learning community. For example, user can easily replace the model definition in Python with KoKOYT-generated codes, while still utilizing the rich code base from the Python community for data preparation, model tuning, serving and visualization, as all the examples in the supplementary do. KOKOYι itself leverages compilers specialized for Python deep learning program such as TorchSeript (PyTorch 202T to further accelerate the generated code.
The design of KOKOYI compiler is covered in Section 4 model is written out entirely from single sample point of view, but processes batches of them after compilation (see Section
KOKOYι performs auto-batching where the
3 PROGRAMMING IN KOKOYI
Our goal is to design KOKOYI in such a way that it requires almost no learning for rescarchers who have written papers. In sense, kokoyi-land is dialect of LSTEX. KOKOYI can typeset mathematics written kokoyi-lans as they were written in LЯTX.. All mathematics this section typeset, parsed, and compiled by KOKOYI, thereby removing the gap between math-in- paper and math-in-code. kokoyi-lang is also more than a ETmX dialect. Mathematics written in kokoyi-lee -lang are executable programs carrying the semanties that most members of the DL community will reasonably expect when reading them.
The immediate challenge face is that math formulations immensely flexible. Repurposing LTTX as the programming interface must therefore allow practical and efficient code generation without losing expressiveness. In contrast to programming languages that require alphanumeric variable names, kokoyi-lang allows programmers to denote variables by math symbols typeset in LTTEX. The "real estate" of the sub- and superscription needs particular attention. After several iterations, we decide to leave them as part of variable name and impose reasonable constraints elsewhere. For instance, we choose [x; 0) instead of fa(x) for learnable functions (i.e. modules), and x [i] (typeset as #zg) for indexing. We also introduce common Python syntax where there is no ambiguity such as raising a power and various forms of multiplication, see supplementary for concrete examples.
We now proceed to introduce kokoyi Sackoyi-langii lang's key constructs: tensors, functions and modules.
Tensor definition A key construct in kokoyi - -lnng is tensor definition, which enables one to define (+) a tensor as the value of an expression, with the command \gets:
[tensor] + [expression]
Expressions in tensor definitions be applications of built-in tensor operations. such as transpose ("), element-wing wise arithmetic (including dlement-wise reductions such as sum Σ and product ID] dot product (.), and indexing (-13)-
3

---

Under review as a conference paper at ICLR 2022
The expression in tensor definition can also be tensor constructor, which gives entry-by-ente definition of a tensor. For example, one can define the M x N attention matrix in Transformer (Vaswani et al. 2017)
q Va
Kb]
N
M
qh kbp vd
)}
softmax
softmax
i=0
1=0
<=0
where d is embedding dimension, and q and are M x d and N x d matrices consisting of query and key vectors, respectively. The one on the left is full self-attention whereas the right is auto-regressive, iee only attentive to tokens hehind the current one; a sequence-fo-sequenee model, the first is often used in encoder and the second in decoder. One similarly define multi-head attention matrix, by adding another index that ranges over [0, H),, where H is the number of heads (see supplementary for the full Transformer model).
One can also define a tensor in kokoyi miii-lamn -lang by giving a recurrence relation among its parts, such as rows and columns. For example, the hidden state matrix of vanilla recurrent neural network (RNN) can be given by
-1h Sho MOSISL) + (tanh(W, hic-11 + W₂ 2[(] + b)
t= 0
(2)
otherwise
where x is a matrix with L rows, each being the embedding of an input token, and W,, W., and ho are learnable parameters. This is an example of KOKOYI module that we will describe next. To facilitate compiler optimization, programmers need to annotate indices where recurrence relations exist with range (e.g. hinsesr)), which can depend on the dimensions of other tensors as in the case of tensor constructor. Also notice the use of \begin cases) } and \end ( (cases) (rendered as the big left brace in Eq. 2 for branching (Fig. 1b)
is common for a recurrence relation to involve more than one tensor. For example, the hidden state and memory cell of a long short-term memory (LSTM) network (Hochreiter & Schmidhuber 1997) depends on the network's 's forget, input, and output gates, which in turn depends on the previous hidden state and memory cell. This recursive mutual dependency can be expressed in kokoyi 10--1aa ang as
fnsis + o(U h + Vy Iig + by) [[sest] + oU, h Vo m/p] + bo) Co
ipsesz + oU, h h+ V [11] + b.)
q[stst) < oU, h + Vo 214 + ba) ho
t=
t= 0
Museeu + (op tanh(c otherwise
Uig 0p in cωp otherwise
Function and module The syntax for tensor definition applies to function definitions as well. The following is definition of the sigmoid function:
(x) 1
Bokoyi-lang borrows the concept of module from popular DL frameworks such as Py PyTorch (Paszke al. (019)) to enable encapsulation of learnable parameters. Conceptually, a module is function parameterized by learnable parameters that are updated iteratively, for example, by gradient descent. Consistent with the DL practice, kokoyi Setoyi-lac -lang separates learnable parameters from parameters that the function can be applied to using semicolon: f(x; 0). Similarly, if f uses component module g that itself contains learnable parameters, it is written as f(;;)) g). The syntax of module definition resembles those of ыТЕX packages for pseudocode.
Fig. bb) demonstrates the major concepts of tokoyi-lane hidden representation hur. embeds the input sentence S that contains the indices of embedding of the tokens. Wz, Wn, b, ho and the embedding table 1) are all trainable parameters. Note that |S] is syntactic sugar to retrieve the first dimension of a multi-dimennional tensor, whereas the more general GetShape function returns the complete shape. {0) combines a tensor construction (of a zero input) and concatenation right-shift the input. The rest of the codes are all self-explanatory.
with a L-layer RNN, where the final
4

---

Under review as a conference paper at ICLR 2022
c==w
(variable)
0, 02, -2,1,.0.... 1.....
(constant)
let v in
(let binding)
letrec . ine
(letrec binding)
A u.).
(function abstraction)
(((e.....,uu) U&) if e then e1 else
(function application) (branch)
> [m1 Su S v2l. vmap v on
(functional array)
(vectorized map).
Figure 2: Grammar KOKOYI IR. e and crepresents expression and variable, respectively.
4 THE KOKOYI COMPILER
Intermediate representation specialized for deep learning Deep learning compilers have been popular research topic in the recent years which leaves several choices of Intermediate Representation (IR) for our compiler. Among them, Relay (Roesch et al., 2018) uses functional IR due to its power in expressing function closure; TorchScript (PyTorch 2021 uses Static Single Assignment (SSA) for its simplicity; JAX (Bradbury et al. 2018) relies on XLA which translates user program first into High Level Optimizer (HLO) IR for deep learning specific optimizations and then lowers it to LLVM IR.
The KOKOYI compiler chooses functional IR because the mathematical language is inherently very functional, e.g., functions are pure with no side-effect, there is almost no symbol redefinition, etc. This makes it easy for KOKOYI compiler to translate a program into functional IR by traversing the abstract syntax tree and applying syntactic rules. The full grammar of KOKOYT IR is in Figure 2 Generally speaking, KOKOYI IR follows the A-normal form with two additional rules to capture the common loop patterns in deep learning programs, namely the vectorized map (as in Transformer Eq. I and functional array (as in RNN (D). Having these patterns IR makes it very convenient to perform optimizations them later. They demonstrate the necessity of lifting the program interface to math-like language. While some tools have identified the importance these patterns and have provided specialized Python APIs for them (e.g., jax. vmap, jax. lax. scan), they typically require complicated specifications from users due to Python's 's notorious versatility. By contrast, expressing them in math equations is almost out natural instinct which KOKOYI utilizes to achieve both high usability and efficiency. The translated IR of the attention module is shown in Figure c(...
Strong type or weak type? The choice of type system has a significant impact on a programming language and its compiler. The debate whether language should embrace stronger or weaker type system prolongs into the era of deep learning. On one hand, Python's 's dynamically-typed design saves users from annotating types of variables and functions, which boosts the productivity of model development. On the other hand, lacking static type checking leads to elusive bugs, causing issues like unrunnable codes, buggy codes and reproducibility. Furthermore, deep learning compilers continuously push for stronger type semantics to achieve high performance. For example, TorchScript's script mode requires users provide type hints for the argument and return values of every method. For more complicated programs, it also provides a trace mode that executes the program once and records the data flow well as the runtime type information for further optimizations. Program tracing cannot deal with control flows that may diverge during runtime. To circumvent this, AutoGraph (Moldovan et al. 2018) (which is integrated into tensorf] ow funct on) adopts source code transformation to convert control flows to data flow operators. Nevertheless, the transformation rules are limited and can fail in certain cases.
KOKOYI'S principle is usability oriented. As such, no type hint is required by default. KOKOYI tries to infer the expression type from intrinsic language semantics. Some operators (e.g., matmul) have constraints on the types of inputs and outputs they be applied to (e.g., tensors). Similarly, the two new IR expressions introduced by KOKOYI, functional array and vectorized map. have list and tensor type respectively. We propagate these type constraints among the IR nodes to refine
5

---

Under review as a conference paper at ICLR 2022
Attn((o) ((1)
[K)MM
lotrec Attn À 9, k, d.
28length(a) in
L/--h kbl
M
let N (8longth(k) in
softmax
(va Vd
let 31 MKarange(O, M) in
f=0
i=0
let X3 = Maraange N) in
(a)
letrec 2.
1
def Attn(a, K, d):
batch scope:
let x5 MMindex get(q, i) in
Bockoyi.lengthn
let 3Stranspose(85)
3
(k)
let %7 = KSindex get 5) in
4
kokoyi. arange (0, M)
K8natmull (X6, X7) in
2(1):
batch scope: 00 kokoyi arange N)
let X9 = KNsgrt (d) in
6
X10 XN8iv(NS %9)
4()
batch scope: 04,
in X10
8
5
q[á]
9
6 kokoy1. transpooso_55)
let X11 = vaap X4 on X3 in let X12 #Xsoftmaxx %12
k[5l
11
8 kokoy1..na mat.ulllnccc.mmmmmmmm (_,, _7)
12
kokoyi. kokoyi.sgrt(d) sqrt(d)
in
10
let X13
vmap 32 on
14
return _0
_11 kokoyı, vnap _3)
%13
16
return kokeyi. softnox 11)
in
17
return kokoyi. vmap )
in Attn
(b)
(c)
Figure 3: (a) Attention module in kokoyi-lang; (b) Python code generated by KOKOYt; (c) Intermediate representation of the program.
types that are unknown. Also note that is not uncommon to see type notations when describing a model during paper writing. Examples are æ IR xd2 to specify a matrix-type variable and f: (R&4, Rd)) + Rdh xda to specify a function. extra type constraints in the future, which will also help to aato-geenrrate more initialization codes.
straightforward for KOкOYI parse them into
The viable types in KOKOYI are scalar (i.e., integer or floating number), list, tensor, function and tuple. To support type polymorphism (e.g., x + can be between two scalars, or two tensors, or one scalar and one tensor), KOKOYT iteratively refines expression types by incorporating more precise type constraints. Specifically, the first pass only collects the most general types of cach operator, type hints from user annotations as well as from IR requirements. Once the constraints are resolved, we refine the type of each operator according to their signatures, which will introduce more type constraints for further refinement. Type constraints are collected using the let-pollmorphism (Milner) 1978) algorithm and are resolved by unification. If an expression type cannot be determined statically, KOKOYT will generate code with the most general type and defer the type dispatch to runtime. Otherwise, KOKOYI will generate more precise implementation. For instance, although both RNN (below left) and ReLU (below right) uses the case expression to represent branching, KOKOYι can generate if. else... statement for RNN while the vectorized torch. where operator for ReLU in most cases.
|1S1E11 &- ho on tanh(cg)
== otherwise
ReLU(()) {or
x x
otherwise
5 THE AUTO-BATCHING RUNTIME
KOKOYI promotes the concept of "think in one sample, run in multiple". Users express model as parameterized function (or a module) over one input sample, from which KOKOYt automatically converts to an implementation that executes multiple samples in batch. The need for auto-batching not only appears at the scope of an entire program but also at the level of one statement one expression, or even multiple places that are nested. The code to compute attention Eq. | shows such example. It calculates attentions between all positions of array and by nesting two array definitions and one will wish to compute them in one batch. Morcover, the inner tensor expression can have dynamic length shown in the decoder attention example.

---

Under review as a conference paper at ICLR 2022
Another design consideration is debuggability. The best way to avoid bugs is to not having them in the first place, expressing a model in equations already helps to some extent, as we experienced ourselves in prototyping models in KOKOYI. Nevertheless, we cannot realistically hope no bugs creep into the implementation. The success of PyTorch has proven that the run-and-see experience is favored by large population of users, which we wish keep in KOKOYT. Unfortunately, none of the previous auto-batching solutions can satisfy all the requirements.
In KOKOYI, we build runtime system that supports nested, dynamic and eagerly executed auto- batching on the top of tensor-based deep learning framework (e.g., PyTorch). Our design is centered around an operational semantics called General Universal Function (GUFunc) which specifies how a function over one sample can be generalized to multi-sample inputs.
Operational semantics GUFundlis broadly adopted by NumPy to generalize functions originally over scalar elements to sub-arrays. A GUFune specifies the element shape, called core dimensions, of its arguments and return values. For example, matrix multiplication is annotated as (a, b), (b, c) -> (a, c) meaning that the element shape of the two arguments and the return value is matrix. To support batch processing, a GUFunnc allows prepending arbitrary number of batch dimensions its arguments as long as the dimensions satisfy the broadcasting semantics P For example. NumPy's matmul supports inputs of shapes (32, 1, 10, 5) and (4, 5, 6) respectively because the element shapes (underlined) are valid for matrix multiplication and the batch dimensions (32, 1) and (4) are broadcastable. Conceptually, one kernel conducts 32 x 4 = 128 multiplications.
Hence, the auto-batching problem becomes how to make the entire program a GUFune where its core dimensions specify the shape of an individual sample. To achieve that, KOKOYI first needs to know the number of batch and core dimensions. At the moment, we ask users to provide such information function while leaving automatic inference as future work.
when invoking a KOKOYI-generan KOKOYI then ensures that each invoked operator satisfies the GUFunc property with regard to the batch and core dimensionality passed on during execution. For most operators, this is as most PyTorch operators have supported broadcasting. For operators working along a certain axis such reduction normalization (e.g.. \softmax), KOKOYI adjusts the working axis according to the batch and core dimensionality. Since the GUFunc property is composable (a function consisting of only GUFunes is GUFunc), the entire program can execute all samples in one batch.
Eager-mode vectorized map KOKOYT's vmap (f, x) is high-order function with its first argument being function and the second one being tensor. Conceptually, it applies f over each element of x and returns the transformed results in a batched tensor. We call f the mapped function and x the mapped domuin. The mapped function acts as a transformation routine over one element. (Line 5-16) computes for one position the
As is shown in Figure3]b,, the mapped function attention against all positions j. The vmap call (Line 17) applics it over domain variable _3 which is calculated by arange (0, M) (Line 4). This function further contains another nested function. that is mapped by the vmap at Line 15, over the mapped domain a arange (0, N).
During execution, KOKOYI maintains global batching scope. Initially, the batching scope has shape (), meaning no batch dimensions. vmap lifts the current batching scope by appending the batch size of the mapped domain. Whenever tensor created outside of the current batching scope is referenced inside a mapped function, its batching dimension is expanded by appending the batch size of the current scope. For instance, in Figure a)), scalar variable is created at Line outside of any mapped functions. When it is referenced at Line 6 by kokoyi. arange, it is automatically expanded a vector of shape (M, ). Because all KOKOYI operators are GUFunes, they automatically work on batch of inputs so the computation is vectorized. One subtlety is that the vmap itself must be a GUFunc too to allow nested invocation. This is guaranteed by appending the batch size so that the original batch dimensions are retained.
To support dynamically shaped batching as in the decoder attention module, KOKOYI associates each tensor variable with a mask tensor, boolean tensor indicating whether the corresponding entry is valid. Masks are created by operators that may lead to dynamic shapes within a batch (c.g., kokoyi.arange) KOKOYI operators will utilize the mask tensor during computation, e.g.. skipping invalid entries during reduction, calculating the result mask or simply dropping it.
attos: http:////mm...eeee...eeaaaeeeeee
html
http://wwww.ar////////////aa/l/s Ing.

---

Under review as a conference paper at ICLR 2022
Naive runtime bato-bbtchiig can lead to several performance issues. First of all, expanding batch dimensions may duplicate tensor data. To give an example, suppose matrix ς (Figure (()),, Line 1) has shape (M, D), referencing it inside the nested mapped function (Line 8) expands it to be (M, N, M, D) where each (M, D) sub-tensors are the same. Our implementation resolves the problem by taking advantage of the broadcasting semanties. We unsqueeze q's dimension 1, M, D) and let broadcasting take affect during subsequent operations. Another issue is with the index lookup q q[i] (Line 8). Although the lookup vectorized for every element in i, the particular case has more optimized solution using tensor slicing [0 [O:M]. We hereby implement compiler optimization pass that makes such transformation automatically when detecting the index variable being a range domain.
(1,
EVALUATION
Diversity and Coverage To measure the flexibility of kokoyi-lang, we have implemented
variety of popular models, including Multi-layer Perceptron (MLP), LeNet (LeCun et al.) [998).
UNet (Ronneberger et al.) (0155) for images segmentation, sequence models for sentimental classi-
fication, neural machine translation (with and without attention (Sutskever et al. 2014)) using
LSTM (Hochreiter & Schmidhuber] 1997). Bi-LSTM (Huang et al.] 2015), and finally Trans- former (Vaswani et al. 2017). For generative model, we implemented Generative Adversarial Networks (GAN (Goodfellow et al. 2014) and Variational Auto-Encoder (VAE (Kingma & Welling 2013)). Finally, in the context of reinforcement learning, we implemented Policy Gradient (PG) and Deep Q-learing Networks (DQN). We have included HTML files of these notebooks in the supplementary.
1.25
kokoyf
ttrch
cokey
1,01
0.94 0.94
1.02
0.95
Relative Training Speed
0.75
Batch size
120
691
1K
3.59
4.04
758
185
0.00
18
Lohal
LoTd
BLSTMM
D4H
Transfermee
512
372
375
100
200
300
400
Training time per-iteration (ms)
Training time par-iteration (ms)
Figure 4: Relative training speed V.s. PyTorch
Figure 5: Per-iteration training time under different batch sizes for MLP (left) and Transformer (right).
Performance We tested KokOYt's system efficiency on a variety of deep neural network models and compared them with the state-of-the-art well-optimized implementations in native Py (see Appendix C for the model details). We measured the running time of training these models for one iteration, including forward and backward propagation pass and the parameter update, and reported the average time of 20 runs. We observed little variance. All the experiments are conducted on an AWS EC2 g4dn.2xlarge instance equipped with 16GB NVIDIA T4 GPU card. The software used are PyTorch 1.00.. and CUDA 11.0.
The training speed of these models generally match their PyTorch versions (< 6%) as shown in Figure 4 and is stable with larger batch sizes (as in Figure 5). showing the effectiveness of KOKOYT's auto-batching.
Usability Coding math equations is concise and succinet, especially when the model is compli- cated. For example, Transformer in kokoyi - -ang has only 62 lines of code, compared with 155 in the Py ToTchh implementation (we count only the forward functions). The key that KOKOYI codes much compact has to do with the fact that it specifies the computation at finer granularity and lets auto-batching handles batching, whereas the Py Torch implementation must make sure all operators are batched and aligned, Appendix for detailed inspection.
8

---

Under review as a conference paper at ICLR 2022
Model
LoC (KOKOYT)
LoC (PyTorch)
MLP
10
10
LSTM
25
22
Seq2Seq
30
34
Transformer
62
155
Table I: Comparison on line-of-code (LoC) between KOKOYI and PyTorch
RELATED WORK
Compilers for deep learning frameworks are generally divided into two categories: Tensor compilers such Halide (Ragan- (Ragan-Kelley et al., 2013). TVM (Chen al. 2018) and Tensor Com- prehension /Vasilache et al. 2018) aims at generating efficient codes for tensor operation; Program translators like AutoGraph (Molovvan et al. 2018) and JAX (Bradbury et al. 2018) converts Python into low-level IRs for more efficient execution. KOKOYI can leverage both of them: speeding up individual tensor operator by plugging in functions produced by the tensor compilers; replacing the code generation stage and the runtime with the program translator's for faster overall execution.
Auto-batching is a technique that transforms the computation for single instance to the one for batch. TensorFlow-Fod (Looks et al. (017)) provides domain-specific languages for users to express sample-win wise computation and performs batching the computation graph, but has limited support for control flow. DyNet (Neubig et al.) 2017) generates and batches computation graphs dynamically, but with significant runtime overhead. DGL (Wang al. 2019) utilizes domain knowledge from the graph neural network family for auto-batching, but cannot generalize to arbitrary programs. (Radul al. 2019) proposed local static auto-batching solution that could handle control-flow intensive programs. None of these solutions can handle dynamic shapes.
FUTURE WORK
There are many possible improvements to the current KOKOYI prototype, including:
A better cokoyi-llnm editor. Users of kokoyi-lang may frequeetly write mathematical notations that are compact after rendering but cumbersome to 18l,,, which necessitates a better editor. There have been many efforts in developing What You See Is What You Mean (WYS/WYM) editors for mathematical equations and even ETbX (Kastrup) 2002), which we believe are good bases to start with. Early users have also expressed making KOKOYI available in other IDEs.
Effective debugging tools. Another question is how to present debugging information. Compilers the locations source code.
for traditional programming languages associate error messages with As for KOKOYI, the error messages must be further rendered alongside the highly compact symbol definitions, since the generated backend code has less readability.
More compiler optimizations. Many traditional code optimizations for the functional language family are applicable Eokoyi-lamd including control-fice flow analysis, lambda lifting, inlining and so on. Recent advancements such as kernel fusion (Chen et al.) (2018) and graph substitution (Jial ct al. 2019) that optimize the computation graph of deep learning programs are also promising to integrate.
Towards a web of executable papers. Our ultimate vision is to build an ecosystem where papers are executable. This can be accomplished by supporting compilation inside ETTFX documents. From that perspective, a research paper is nothing more than elaborated documentation explaining an executable model. Since these snippets are executable, they are guaranteed to be syntactically correct, removing the error-prone process reproduce their correct implementations. In fact, snippets of KOKOYI codes in different research papers can be gross-linked. making them reusable for the wide community consume.
KOKOYI code snippets

---

Under review as a conference paper at ICLR 2022
REFERENCES
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclau- rin, and Skye Wanderman-Miine JAX: composable transformations of Python+NumPy programs, 2018. http::///ii http//ttttttbttt.g//g com heam:/googee/aai
Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang. Yuwei Hu, Luis Ceze, et al. (TVM): : An automated end-to-end optimizing compiler for deep learning. In 13th {USENIX} Symposium on Operating Systems Design and Implementation (OSD)) 18), pp. 578-594, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1788 1997.
Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional Istm-crf models for sequence tagging. arXiv preprint arXiv:/508. 01991, 2015.
Zhihao Jia, Oded Padon, James Thomas, Todd Warszawski, Matei Zaharia, and Alex Aiken. Taso: optimizing deep learning computation with automatic generation of graph substitutions. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pp. 47-62, 2019.
David Kastrup. Revisiting wysiwyg paradigms for authoring latex. COMMUNICATIONS OF THE TEX USERS GROUP TUGBOAT EDITOR BARBARA BEETON PROCEEDINGS EDITORS KAJA CHRISTIANSEN, 23(1):57, 2002.
Diederik P Kingma and Max Welling. arXiv:1312.61 14, 2013.
Auto-encoding variational bayes. arXiv preprint
Leslie Lamport. LATEX: a document preparation system: user's guide and reference manual. Addison-wesley, 1994.
Chris Lattner and Vikram Adve. Llvm: A compilation framework for lifelong program analysis transformation. In International Symposium Code Generation and Optimization, 2004. CGO 2004., pp. 75-86. IEEE, 2004.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (1((1)):278-2222 1998.
Moshe Looks, Marcello Herreshoff, DeLesley Hutchins, and Peter Norvig. Deep learning with dynamic computation graphs. arXiv preprint 170.0017,,,,,,81, 2017.
Robin Milner. A theory of type polymorphism in programming. Journal of computer and system sciences, 17(3):348-377 1978.
Dan Moldovan, James M Decker, Fei Wang, Andrew A Johnson, Brian K Lee, Zachary Nado, D Sculley, Tiark Rompf, and Alexander Wiltschko. Autograph: Imperative-stie coding with graph-based performance. arXiv preprint arXiv:1810.08061 2018.
Graham Neubig, Yoav Goldberg, and Chris Dyer. On-the-fly operation batching in dynamic computa- tion graphs. In Advances in Neural Information Processing Systems, pp. 3971-3981, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luea Antiga, et al. Pytorch: An imperative style, high-perfie bigh-performance deep learning library. In Advances Neural Information Processing Systems, pp. 8024-8035. 2019.
PyTorch. Torchscript, 2021. URL http://///pororc....ooo/////aa/a
Alexey Radul, Brian Patton, Dougal Maclaurin, Matthew D Hoffman, and Rif A Saurous. Automatically batching control-intent intensl-nteensive programs for modern accelerators. arXiv preprint arXiv: 1144,, 2019.
10

---

Under review as a conference paper at ICLR 2022
Jonathan Bagan-Kelley Connelly Barnes, Andrew Adams, Sylvain Paris, Frédo Durand, and Saman Amarasinghe. Halide: a language and compiler for optimizing parallelism, locality, and recompu- tation in image processing pipelines. Acm Sigplan Notices, 48(6):519-530, 2013.
Jared Roesch, Steven Lyubomirsky, Logan Weber, Josh Pollock, Marisa Kirisame, Tianqi Chen, and Zachary Tatlock. Relay: A new ir for machine learning frameworks. In Proceedings of the 2nd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pp- 58-68, 68, 2018.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical image computing and computer- assisted intervention, pp. 234-241 Springer. 2015.
lly Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-3112, 2014.
Nicolas Vasilache, Oleksandr Zinenko, Theodoros Theodoridis, Priya Goyal, Zachary DeVito, William S Moses, Sven Verdoolaege, Andrew Adams, and Albert Cohen. Tensor comprehen- sions: Framework-agnoose high-performance machine learning abstractions. arXiv preprint arXiv:1802.04730, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 6008, 2017.
Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma. Lingfan Yu, Yu Gai, al. Deep graph library: A graph-centric, highly-performant package for 2019.
graph neural networks. arXiv preprint

---

Under review as a conference paper at ICLR 2022
A ESTIMATION OF KOKOYT'S APPLICABILITY
KOKOYt as stands can mend the gap between math-in-codes and math-in-pape and help researchers who focus new model development. Of course, not all research fall into such category. We sampled papers from five top machine learning conferences and classified them into the following categories:
Model-NN: Papers that proposed new model architectures, new loss functions or new neural network building blocks.
2. Algorithm: Papers that proposed new algorithm/models but they generally are not imple- mented with neural networks.
3. Optim: Papers with focus on optimization techniques.
4. Theory: Papers about theoretical analysis.
5. Other: Papers not in the above categories.
6 shows the result. KOKOYI can potentially benefit most papers in the Model-NN category, ranging from 15% to 66%.
Ohhr 15.4%
ModssNNN 15.05
Ooli 2.8%
Alyonitmm
55.4%
(a) ICLR 2020
Other
Masis-NN
ModelANN
Theary
Theoy
Opdm 12.0%
Algorthmm
Algorifm 25.6%
55.5%
(b) ICML 2019
(c) NIPS 2019
Other 22.2%
Othe
33.3%
Theery
2.2% Aggritirs 8.3%
ModskNNN
66.7%
(d) ACL 2019
(e) CVPR 2019
Figure 6: Classification of the papers sampled from several recent conferences
12

---

Under review as a conference paper at ICLR 2022
B STUDY OF TRANSFORMER CODE
To quantify the actual Line-of-code saving of Kokoyi on the Transformer model, we compare the implementation PyTorch and KOKOYI side-by-side in Figure 7 Overall, large portion of the Py Torch implementation is for manipulating shapes of the tensors to align with the batching requirement of the subsequent operations. By contrast, these non-math parts are completely absent in the KOKOYI implementation thanks to the flexible and succinct tensor constructor syntax. Yet, KOKOYI can still achieve similar training speed by its auto-batching design.
Mytorchiltte(,, di 0. Chapel 11
+Js
(batch_.azze
Swap sxis that attention can be computed between all pairs
Fumction Atta((()* (.00)
Sranappae 44 :3)
scores forch, astte Its, sath.sort(()) Fasoftn
<- ltyl
(A,
Specifyy axis
de Ay 13 12
Retarn
10
duf Pytorci Pytoocieesuulatinn K2 +): a u. Ru vt (betcflus er mend meeeenneeee freathy serchhyy teskierztine) Swap axis so that attention can be computed between pairs
shape
Mukockitullee/ (A9", (+f"")
cart(d)
woOe Trrch nit (hiTEho hineeiinne (Ta(erwe)))
d + ldal
acures
mask
Calculate mask
16
a cortennn(saseedd crores, Idiewd
[Selinass Eaallli
17
rut worch (e, x) neturn not
18
Specify axis
Returu a-e
class PyTorckiiitttt
Noduie)) snct (seit, iip, hidden_din, honds, outpettition, is nower)) cotr ()
22
21
Modale MIlArta((e)) t (A))
34
oelf ALa ttrrh. nn. Paranetar (torch hanrrnn inpvi_ hénden_din))
onl tarch mn.Faranetertten Teereh-TTeesereee (nue. heats, topur zin, hidden, dia)) self. lursh Poraeceerrrru TaarchhTTasrrennn heall,, dia, (isl)e) sutput_din))
L [AP9)
aew.Peraneterttrre hmvertunn Is.wwsk
we- (atti(> Wr. a. Meg. Mgglss 0yaal" Radurn max 11
30
sely. Input ein = trput dia o output_dix autpit_din self leads nue heads only hiMMontts aiddon_tte
Reshape the weight matrices to align with the shape requirement batched matrix multiplication.
31
36
tonuard(ents, by: by (botch
input die)
KLa 20l/ Urwnogusee inallppe (ael) incsiow, snl] mun Jeals self kiddlen simw it cole-u M tranepote (. rechape encr in cite. eeir nue haade cate- badden csse) transpenn recupnteel dixe,
37
corch.nstmi(ss W.5)
torch (0, K.K)
12
(brch.neteuik,, v)
Recover the num, heads dimension.
qi shapsto). < sh aetf-nun nue huads, self Hidcer aics)
chapet
mapeet colp.nue heats,
48
transeseth C
Swap axis so that the follow-up attention can be computed for multiple heeds at once.
49
k (ranspone 2)
v. t.tanspesee).
alees
55
all Allde, k v)
Recover the manipulated shapes and align with the shape requirement the matrix multiplicaiion
56
alti all. transpon())))
reshh.resh (ett1 shapefol, att3, shopett), 4))
terch,,atwoleett,, sel)
return
Figure 7: Comparison of the multi-head attention code in PyTorc and KOKOYL. We hightlight the parts of shape manipulation that are saved by KOKOYI.
13

---

Under review as a conference paper at ICLR 2022
C MODEL BENCHMARK DETAILS
Synthetic inputs are used for measuring training speed. For all the models, we have verified their accuracy on real datasets (see the supplementary). Below lists the model configurations in Figure 4
For MLP, we use input size of 784, batch size of 512 and all the hidden sizes are also 512.
For LeNet, we use an input image size of 32 x 32 with one channel, batch size of 64 and all the hidden feature maps have 512 channels.
input sequence of length 50, word vocabulary of size 5000 and a batch size of 512. Both the word embedding size and hidden sizes are set to 512.
For LSTM and BiLSTM,
For GAN and VAE, we set the size of the latent vector to 512 and test them on input images of size 32 x 32 with one channel. The batch size is 1024.
For Seq2Seq and Transformer, both of their source and target sequences have 50 words drawn from a vocabulary of size 5000. Both the batch size and hidden sizes are set to 512.
14


Reviews:
[{"review_id": "gx4S_V5-pyn", "reviewer": "Reviewer_8Mk2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It is interesting to see a tool that can translate Latex-based math equations/algorithms into Python code. Such a tool could reduce the gap between developing models in math language and implementing them in programming languages. The idea is interesting and the proposed tool is implemented and integrated with Jupyter Notebook.\n\nIt seems that the proposed tool is more useful for deep learning (DL) programs that contain a lot of math equations/computations. The developers still need to write other modules, such as data preparation and result visualization, in Python. For less math-heavy programs, the benefits of the proposed tool are not very clear. It is not clear how many DL programs actually contain many math equations and can be benefited from the proposed tool. Many DL programs can directly reuse existing operators/packages without going to the details of math equations (and they are not written from scratch)?\n\nFurthermore, using the proposed tool, developer still need to write latex code. Although the size of latex code is generally smaller than that of Python code, the efficiency/usability of such a tool still needs to be evaluated. For example, to measure the developer productivity when writing code in Python and in Kokoyi.  Furthermore, like Python code,  Latex code could contain bugs too. The correctness of latex implementation cannot be guaranteed, so there is still a gap between model and its implementation.\n\nThe paper is generally well written.  A minor point: the following two references appear twice:\nMoshe Looks, Marcello Herreshoff, DeLesley Hutchins, and Peter Norvig. Deep learning with\ndynamic computation graphs. arXiv preprint arXiv:1702.02181, 2017a.\n\nGraham Neubig, Yoav Goldberg, and Chris Dyer. On-the-fly operation batching in dynamic computation\ngraphs. In Advances in Neural Information Processing Systems, pp. 3971–3981, 2017a.\n", "summary_of_the_paper": "This paper proposes Kokoyi, which can automatically translate mathematics into Python implementations. The proposed tool consists of kokoyi-lang, a programming language with the syntax of LATEX and the semantics of deep learning mathematics, and kokoyi-lang, a compiler and runtime supporting advanced optimizations. Kokoyi is integrated with Jupyter Notebook. To measure the flexibility of Kokoyi, the authors have implemented a variety of popular DL models and performed evaluation.", "summary_of_the_review": "Pros:\n. The proposed idea is interesting.\n. The proposed tool is implemented and integrated with Jupyter Notebook. \n\nCons:\n. The benefits of the proposed tool are unclear. \n. The evaluation is limited.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FSma9U0n6lk", "reviewer": "Reviewer_aMmp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\nThe idea of executable papers is interesting, and supporting auto-batching in model implementation languages is a significant problem. The authors present a nice algorithm for the auto-batching problem that has less overhead. \n \n**Weaknesses**\n\nWhile I understand the appeal to have a LaTeX-based coding language to enable executable papers, I do not feel that it is easier to code in LaTeX rather than code. For most parts, the LaTeX code looks like a fancy new syntax for the actual code (e.g. summations instead of loops), but with several limitations: (i) the lack of state (which the authors also point out) (ii) lack of abstractions, modularity, objects, etc. and (iii) inability to partially evaluate the model (for debugging, again the authors identify this issue). These issues both make it hard for a user to program in kokoyi and might result in inefficient code. For example, since there is no state in the loops (summation),  it is not possible to merge multiple data-processing steps into one loop. \n\nThe authors claim that the main benefit is the ability to support auto-batching. While auto-batching is great, it is unclear why it can not be done inside a language like PyTorch with some special syntax (similar to how NumPy does function vectorization). What exactly does a language like LaTeX provide here that facilitate easy auto-batching?\n\n\nThe evaluation is great, but it is missing several details. \n — Why does figure 4 only report bars for 5 models? How does Bi-LSTM, GANs, VAE, PG, and DQN perform?\n\n—What were these models trained on? What is the size of the model, size of the dataset, etc.?\n\n— How does memory usage look like for the two implementations? Is there extra memory overhead because of the translation?\n\n— How many times were the results in Figures 4 and 5 run? What is the standard deviation?\n\n— Table 1 does not report the numbers for all the models.\n\n— A user study evaluating the ease of use of kokoyi would answer whether it is easy to code in python or LaTeX?\n", "summary_of_the_paper": "The paper presents a LaTeX-based language and compiler called kokoyi to write math-based models and compile them to actual code (such as PyTorch). The authors present an approach to support optimizations such as auto-batching during this compilation process which significantly reduces user burden. The authors presented kokoyi implementations of several popular models such MLP, CNNs, LSTMs and transformers and showed that the kokoyi compiler does not introduce much performance drop. ", "summary_of_the_review": "I lean towards rejecting this paper. The authors didn’t make a compelling case for why they need a new LaTeX-based language to enable auto-batching. The evaluation also lacks several details. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "A1DpzuoQck", "reviewer": "Reviewer_cySy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and clear. It hides many aspects of translation and focuses more on the user side. The idea is extremely fascinating, especially about  \"executable papers\" which would be very interesting.\nUnfortunately, the approach is not testable. I would have appreciated the possibility to test the system to understand how much the language helps to implement my network.\nIf I have to find a second weakness, probably it is that the paper does not provide formal definitions or details about the implementation. But the message the authors wanted to give is different, thus I cannot consider this as a malus.\nHonestly, as said above, I do not see any problem with the paper itself. It is discursive, it tells a story, which in my opinion is a good story, worth discussing. However, it is a story that I have to trust but that cannot play with, which limits the quality of the article.\n\n", "summary_of_the_paper": "The paper presents a language similar to LaTex with a compiler able to translate mathematical formulas of network layers into executable source code. The article considered the PyTorch framework and presented several usability and performance studies.", "summary_of_the_review": "The paper is good. However, it presents a system that cannot be tested.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes that ML models might be better expressed in a way closer to their mathematical representation than to Python code.  This is an attractive proposition, but the paper's development of this proposition is that models might best be expressed in LaTeX, which is not a hypothesis that the reviewers consider proven.

Ultimately, this paper proposes a new language in which to express ML models, and compares that language against one baseline: PyTorch.  However, this is far from a reasonable baseline.  Even within Python, systems such as JAX (which the paper dismisses as a "Program translator") are much closer to the pure functional style; and going further afield, comparisons should be to DEX and Julia, to name but two.

The reviewers appreciate the approach to autobroadcasting, but again note that this does not require a new language, and again, systems like JAX, DEX, Julia all have approaches to broadcasting which are not compared.

Reviewer 8MK2 is concerned that we "still need to write other modules .. in Python", but the authors rebut this well: Kokoyi is not expected to be applied to an entire program, just to the model components.

Even if Kokoyi were to be successful, there is a question of its wider applicability.  A major strength of PyTorch/JAX is that they are used by a much larger community than just ML paper authors.  It is because the authors write in these tools that their work is usable by other practitioners.  The paper explicitly says it is targeted not even at ML paper authors, but at a subset of that community.

The usability analyses are very much lacking.  Lines of code is a notoriously coarse tool to assess programming paradigms.  I would also caution against trying to do any small-group user study - the best initial study is to release Kokoyi into the wild, get feedback from users, and then if it proves popular, prepare a paper or monograph.  This is the path of PyTorch and other frameworks.  

Until then, the paper may be of interest to a workshop very focused on programming models for ML, but is not currently suitable for the wider ICLR audience.
Decision: Reject