Creating A Pluto.jl Notebook Reader In Julia | by Emmett Boudreau | Towards Data Science

Pluto.jl is a rather recent IDE development for processing notebook files in the Julia programming language. While Pluto.jl has some awesome features, the features certainly do not fit my personal preferences — and I am not alone in thinking that. I wrote a whole article detailing my experience with Pluto.jl, what I like about the package and what I dislike, which you may read here:
towardsdatascience.com
Then there was Neptune. Neptune was a fork of Pluto.jl that modified Pluto.jl to exclude all of the interesting things that the original creators intended to be used in every notebook session. I also ended up writing an article about Neptune.jl, which you may read here:
towardsdatascience.com
What I think Pluto.jl and Neptune.jl actually demonstrated to me is that this is very possible. You could have a Julia-based notebook server, and all of the great things that a notebook server offers can be customized and brought forth with the full power of Julia. Regardless of whether or not you use Julia, one must acknowledge that IJulia is likely not the best solution out there to write Julia in. That being said, all of this compounded to make me really want to create my own notebook server, with my own sort of application designed underneath it.
There are a lot of ideas placed on a release canvas for this project, and I have even more in the pipeline for it once the project is actually in a regular development stage and has a solid release out. If you would like to learn more about this new IDE, Jockey.jl, you may check out the Github for the project here:
github.com
One more thing! There was another article where I took an initial look at this format and created some new types to work with this data. That in mind, it might be worth looking into in order to give a bit more context to the overall goal here.
towardsdatascience.com
Before we dive into just making our file reader, we of course need to take a look at how the Pluto.jl file format actually works, I did this in my last article discussing Jockey, and this is what I was greeted with:
Appropriately, my reaction was surprised at how they actually did this. Today we are going to attempt to read this file format and put it into our cell objects. A great thing about a project like this is that it is a lot of debugging, that being said I will be doing this code inside of a notebook. This will make the code I work on a lot easier to share with all my readers — which is an added benefit to things just being easier to explain — as I am not hopping through files. That being said, the notebook is available here:
github.com
[emmac@fedora ~]$ cd dev/nbs && jupyter-notebook
I have Cd’d into our notebook directory, and now will create a new Jupyter session. No need for includes, as the entire goal here is to get these to fit into our types for now. All we need to do is read the lines of this data, and come up with some sort of loading algorithm to transfer this into our new types. The same is also going to be done with .IPYNB. I know, I get it; this is somewhat boring for now, but the result is worth all the work we will do on the back-end.
Anyway, I created a new notebook — and thus, added our first bit of code. This code is just going to be the code from the Jockey Cells.jl file. You may view this file on Github here:
github.com
using UUIDs, TOMLabstract type AbstractCell endmutable struct InputCell <: AbstractCell    ID::UUID    Text::String    env::Dictendmutable struct OutputCell <: AbstractCell    ID::UUID    output::Stringendmutable struct Cell <: AbstractCell    input::Cell    output::Cell    hidden::Bool    lang::Symbol    env::UUIDend
Next, I am going to be reading the file in as a string.
uri = "weird.jl"input = readlines(uri)
The good news is that this string contains a lot of information about both the Pluto.jl file type, and why this type is used. I plan to use some elements of this in my own version, we also need to consider the dependencies that are loaded here. While I am not entirely sure if we are going to need to use interact, as these two servers are very different in nature, we are certainly going to be utilizing this UUID farm. I made a new little function that will allow us to see all of this work in more detail:
function showsome(x::UnitRange, uri::String)    f = readlines(uri)    for i in x       println(f[i])     endend
Using this function, I got a better view of the text in the file. My first test in my approach is going to be writing a function that will get us all of the UUIDs, this is quite easy I think since they are most certainly noted with a lot of specific characters.
function read_plto(uri::String)    cells = Dict()    celluuids = []    for line in readlines(uri)        if occursin('╟', line)            println(line)        end    endendread_plto(uri)
Nothing crazy, but a step in the right direction. From here, we are going to need to scan for the next cell. Once we hit that cell, we are going to consider that all of the lines between those two cell definitions are going to be cell content. After I get that functionality working, I am going to extract this function out, simply by renaming the method of course, and then write another method which will call it. That will help keep the “ function for step” mentality, and make this package a lot easier to look at. Here is a look at my initial function:
function read_plto(uri::String)    # We need file lines for each cell UUID    cellpos = Dict()    first = 0    ccount = 0    for (count, line) in enumerate(readlines(uri))        if occursin('╟', line)            ccount += 1            if first == 0                first = count            else                push!(cellpos, ccount => first:count)            end           end    end    return(cellpos)end
This for loop is quite gross, but kind of serves as an outline for what we need to do. We can make this look better and change some looping syntax later. Firstly, we are enumerating our read lines with count. The lines are stored as the variable line. Variables first, ccount (cell count), and cellpos (a dictionary) are all initialized prior to this. The first thing we do is see if the line has the char ‘╟’ We saw how this was able to get all of our UUIDs out before. Whenever that line is found, the first thing that will happen is the cell count will populate.
This is ultimately for Jockey, as Jockey will have a key of UUID’s and their respective numbers. However, each cell can be a different number of lines afterwards, which is where the range comes in. The range is going to be a value pair to the cell’s count. Basically I am going to be able to accumulate which lines contain which environment. Let us give this a quick run:
cellpos = read_plto(uri)
Strange.
Firstly, we now know there are 25 cells inside of this notebook file — however it is pretty obvious that the cell ranges were not correct. The first and most obvious error was in how the cell counter was being added to. This needed to be moved into the conditional that is actually counting cells. This then gave me a regular, ordered dictionary that actually had a 1 value and things of that nature. Then is when I realized that the character I had passed was only at the very bottom of the file. Promptly, I changed this to a string that contains the entire beginning. Here is a look at the final cell positioning function:
function read_plto(uri::String)    # We need file lines for each cell UUID    cellpos = Dict()    first = 0    ccount = 0    for (count, line) in enumerate(readlines(uri))        if occursin("# ╔═╡", line)            if first == 0                first = count            else                ccount += 1                push!(cellpos, ccount => first:count)                println(ccount => first:count)            end           end    end    return(cellpos)end
I will rename this to the plto_cell_lines() method, only part of the developer API so nobody will really need to worry about it but me. This will be a blatant usage of the extraction method, which is a method I use to make my code appear more concise and pretty by simply using more functions. I have an entire article on this topic if you are interested, along with some other techniques that do about the same thing. I think these skills are absolutely essential when it comes to Data Science work:
towardsdatascience.com
Now check out what happens whenever I run this function:
cellpos = read_plto(uri)
So close...
Good news is, using the print function I wrote, we can assert that the first value is correct (because it starts with our first cell, ends on the second.):
showsome(17:24, uri)
We need to change a few things, however, because firstly; all of these cells are starting at 17, which kind of defeats the whole purpose here. This just comes down to a lack of reassignment of the first variable after it is used.
if occursin("# ╔═╡", line)            if first == 0                first = count            else                ccount += 1                push!(cellpos, ccount => first:count)                println(ccount => first:count)                first = count            end           end
Another thing I am going to go ahead and do is subtract one from count for the range that gets pushed to the dictionary. This is because we do not want to include the head of the next incoming cell. I am also not sure as to the input/output structure here, as it could be 8 input cells and 8 output cells, or 16 input cells — we are going to figure all of that out once we use this algorithm.
cellpos = plto_cell_lines(uri)Dict{Any, Any} with 35 entries:  5  => 38:40  16 => 82:85  20 => 99:103  35 => 157:161  12 => 64:68  24 => 114:116  28 => 131:133  8  => 51:53  17 => 86:88  30 => 137:139  1  => 17:23  19 => 96:98  22 => 108:110  23 => 111:113  6  => 41:43  32 => 148:150  11 => 61:63  9  => 54:57  31 => 140:147  14 => 72:78  3  => 28:30  29 => 134:136  7  => 44:50  25 => 117:120  33 => 151:153
It seems like this dictionary suddenly has new values. Now let us call this function from within a new function.
function read_plto(uri::String)    end
I also took the liberty to learn a bit more about the cell data by looking at it. I used our good old showsome() method for this. Luckily, it takes a unit range so I was able to view the data by section:
showsome(cellpos[1], uri)# ╔═╡ b129ba7c-953a-11ea-3379-17adae34924cmd"# _Welcome to Pluto!_Pluto is a programming environment for _Julia_, designed to be **interactive** and **helpful**. In this introduction, we will go through the basics of using Pluto. To make it interesting, this notebook does something special: it **changes while you work on it**. Computer magic ✨"showsome(cellpos[2], uri)# ╔═╡ 4d88b926-9543-11ea-293a-1379b1b5ae64md"## CatsLet's say you're like my grandma, and you have a lot of cats. Our story will be about them."
After reviewing this material, I realized that these were all input cells. I think this makes a lot of sense, Pluto.jl notebooks are not normal notebooks — they are active, or “ reactive” rather. I think it is definitely a cool feature and a good implementation, however reactive sessions like this do pose some problems for research. There is a reason why notebooks save output, and that is for reproducible code. I can certainly agree that this can cause problems with state and these sorts of things — but I do not think that eliminating output like this is necessary. With this newfound knowledge of the Pluto.jl data format, we can move forward with our function by calling our new range finder for cell data:
function read_plto(uri::String)    cellpos = plto_cell_lines(uri)    cells = Array{AbstractCell}()    end
The final step in this process is going to be putting all of this data into our Jockey structs appropriately, and also determining what kind of cells we are working with in-line. Fortunately, for this reading we will only have to work with input cells I believe! Let us refer back to our types:
abstract type AbstractCell endmutable struct InputCell <: AbstractCell    ID::UUID    Text::String    env::Dictendmutable struct OutputCell <: AbstractCell    ID::UUID    output::Stringendmutable struct Cell <: AbstractCell    input::AbstractCell    output::AbstractCell    hidden::Bool    lang::Symbol    env::UUIDend
Now we can very easily create our function, which will call the function we just wrote to get the valid line positions, as well as some new functions we will have to write. Take a look:
function read_plto(uri::String)    cellpos = plto_cell_lines(uri)    cells = []    x = readlines(uri)    for cell in values(cellpos)        unprocessed_uuid = x[cell[1]]        text_data = x[cell[2:end]]        identifier = process_uuid(unprocessed_uuid)        inp = InputCell(identifier, text_data)        out = OutputCell(UUIDs.uuid1(), "")        cl = Cell(inp, out, false, :JL, UUIDs.uuid1())        push!(cells, cl)    end    return(cells)end
I believe that this new method will work, we simply need to create a function to process our UUIDs, and then another one to process the text data — which is the harder part. For our first attempt, let us try and get a look at what this UUID is going to actually look like, so we can ensure we will split each string correctly. Also, it sohuld be noted that the Outputs are assigned new IDs, as well.
function process_uuid(text::String)    println(text)    uuid = UUID(id)    return(uuid)end8# ╔═╡ 611c28fa-9542-11ea-1751-fbdedcfb7690
I simply split this string into thirds, grabbed the third element, and then put it into the UUID type.
function process_uuid(text::String)    dims = split(text, ' ')    uuid = UUID(dims[3])    return(uuid)end
I am now realizing that we are actually done. There is no need to write that data algorithm, as ultimately most of that will be handled by Java, and I am going to need to write the parser then anyway. No use parsing the data into Julia just to send it off to JavaScript to become utterly unparsed again. Firstly, I will say — that was a good bit of work but I am glad it is done. The only thing we need to do now is test the code:
read_plto("weird.jl")MethodError: Cannot `convert` an object of type InputCell to an object of type CellClosest candidates are:  convert(::Type{T}, ::T) where T at essentials.jl:205  Cell(::Any, ::Any, ::Any, ::Any, ::Any) at In[1]:15
That is funny, I think that my changes to the types need a kernel restart. Just to be sure that these changes are in here, here are the updated types:
using UUIDs, TOML, Markdownabstract type AbstractCell endmutable struct InputCell <: AbstractCell    ID::UUID    Text::Anyendmutable struct OutputCell <: AbstractCell    ID::UUID    output::Anyendmutable struct Cell <: AbstractCell    input::AbstractCell    output::AbstractCell    hidden::AbstractBool    lang::Symbol    env::UUIDend
These are of course mere skeletons to their future types, but let us fix this issue by rerunning my notebook.
Here we go!
read_plto("weird.jl")
Now that that issue is solved, I have taken a significant step towards getting this package done. However, there is still a substantial amount of work to be done. I am excited to continue blogging about it! Thank you for reading, I hope that maybe this article, given how much cool algorithm stuff is in it was helpful, or at least cool! In the future, I plan to reiterate over these functions and make them a touch smoother, and add some documentation, but for now I am going to bask in the glory that is data loading. Thank you for checking out my article, and happy learning!