andreasgal/pdf.js — Issue #663: Worker implementation overview
Author: jviereck
Created: 2011-10-14 19:49:38+00:00
State: open

Some time back, I've done a very simple implementation of worker support with the hope to speed up rendering. Since then, I've done a few implementations and the current version is available at #635.

That implementation tried to do some stuff at once but it turned out that this is changing too much code at once. Also, as some pieces haven't been clear before starting the implementation, the patches aren't aligned in a reasonable order, such that one could read the first to the 112th and understand all the steps in between.

Over the last few days it turned out that the current code is **not** in a shape to get landed. Therefore I've started to do a new implementation from scratch that does only the essential bits required to get worker support up and running and tries to be as easy to understand as possible in terms of how the patches are structured (do one bit at a time and leave a good bunch of comments in the commit log).

**BUT** before this turns into over 100 patches again, Artur pointed out to do some writeup first on how the final structure will look like and have this as a guiding for the implementation. So here we go!

## Why workers at all?

Rendering a PDF page contains more or less of two steps: 1) Read the PDF data and translate it into a form such that it is easy to draw it and 2) draw the transformed/read data.

Whereas the second step can't be done on a worker right now, as there is no way to draw to a canvas from a worker (at least right now - there is some spec on the way), the first one can.

The time spent on the first part compared to the second differs from PDF page to PDF page, but my guess is that we spend around the same time in each steps.

While we execute on of the steps, the browser UI is not responsive (basically, it's one big while loop, executed until the task is done). Moving the first set of transforming the PDF data into a worker will make this setp **not** block the UI.

## So let's just do it…

Spinning up a new worker from the existing code isn't that complicated - so let's just add some postMessage and onMessage here and there, and we're done - what's the problem???

Well, the *main* problem to the entire implementation is, that using postMessage destroys the shape of objects. Imagine you have some class `Font` that has some properties and some functions you can call on an instance of `Font`. If you post this `Font` object to the main thread, you are left only with the properties of the object, but there is nothing else of the `Font` left - this means, you can't even determ that it was a `Font` (instanceof obj == Font fails!).

That means, whatever we send from the worker, we need to be able to recover the shapes again.

Secondly, once you do a postMessage, the object is gone and it can't fetch any data on the worker anymore. That means, if the `setFont` command on the `CanvasGraphics` want to look up the font object that doesn't work out, as the font isn't processed on the main thread. To solve this, whenever you process a font on the worker thread, you have to save and label it (let's say we call it "Font_A"). In the `setFont` command, instead of looking up the internal object, you have to tell the command to instead lookup for an object labled "Font_A" and use this one then. As the worker send the labeled object "Font_A" to the main thread, the `setFont` command knows what to do.

You might think, that there is a simplier solution then this: as there is an object required and as the main thread has the PDF data, just process all the objects that are required (if `setFont` requires "Font_A", just process the font on the main thread). That's easier, true, but it means you don't have too much of a win by moving things to the worker thread as you again compute a lot of stuff on the main thread which is the point of all this to not do.

## Not all platforms support workers

This all soundes nice and I really want to see worker support enabled, but it's not possible on all browsers. pdf.js depends havely on typed arrays and some browsers don't support sending them to a worker/from a worker using postMessageing. There might be ways around this, but I can expect it gone be way slower then processing on the main thread. Also, some browsers might not support workers. Last, for debugging, it's way easier to have way to turn a switch and debug in a non-worker environment.

Therefore, there has to be some feature detecting if workers should get turned on and if that's not possible have a fallback mechanism to run. The code path between these two things should be as simpilar as possible - in fact, it just should be about spinning up the processing either on the main thread or a worker thread but the basic processing code stays the same.

## Birds eye overview

After this let's get to the real implementation details.

For now assume, worker support is enabled.

Once a new new `PDFDoc` instance is created, a new worker is created that is booted by a new script `pdf_worker_boot.js`. This boot script includes all the pdf.js files and contains the basic logic for handling messages from the main thread.

As the `PDFDoc` instance is created, the PDF stream is send to the worker, which then builds itself an instance of `PDFDoc`. All `PDFDoc` instances created inside a worker have worker support turned off by default as otherwise each worker would just spin a new worker to delegate the work to and that ends up in an endless worker chain.

As the user requests a `Page` from the `PDFDoc` instance using `getPage(N)`, this page gets a `worker` property set, such that the `Page` instance can 'talk' to the worker and make requests.

As a page of the PDFDoc is getting rendered and the PDF page hasen't been processed yet, a message is sent from the page to the worker requesting to process the page. The worker recieves this request and calls `getPage(N)` on its `PDFDoc` instance. Next, the worker calls the `compile` function of the `Page` object to start the processing of the page.

Once the data is processed, the processed that is send back to the main thread. The `PDFDoc` instance there listens for mesages from the worker and as a processed page comes alone, it adds shape to some of the objects (like fonts, images) and calls a function on the page object, that requested the data, that the data is available now and the page can process rendering the page. (In more detail: the page has to create and attach the fonts and load the images first, before starting the actual drawing process.)

For this first implementation, there is only one worker used per PDF file. That way we don't have to deal with any kind of dependecies that can rase as all calls to `postMessage` are async. But as the worker sends back all the data at once and there is only one worker, there is no problem to this.

## Create postMessage-able draw data

As mentioned above, just doing a `postMessage()` call on the objects/commands processed isn't enought, as there is some loose of shape and reference. Most of the internal drawing commands are not affected by this problem, as the arguments passed to the commands have no dependecies. E.g. the `moveTo` command just takes X and Y coordinate, but these are stored already as 'plain' numbers on the `argsArray` as created in the PartialEvaluator (which is the guy that processes the PDF data and comes up with the drawing commands that get drawn in the end).

If a command like the `setFont` command depends on a PDF object, the dependent object is processed and an **IR** (= *Intermedian Representation*) of the object is created, such that is can get send to the main thread using postMessage. The main thread then might have to do some more work to do some basic reconstruting from the IR form. The `setFont` command is changed, such that instead of looking up the object direclty from the PDF data, it looks up the procssed object that was send to the main thread using the IR form. As these IR-processed objects are available on the main thread, the problem of `postMessage()` loosing shapes and references is solved.

The labeling of the objects coresponds to the internal object id within the PDF document.

For simplicity, lets define that ever command the PartialEvaluator creates is in IR form. That means, the `moveTo` command is in IR form as well, although it doesn't require any adjustments as the data is already ready to go.

Once the PartialEvaluator has finished, it has created a list of IR commands, that are bundled in a so called **IRQueue**. As this queue contains only postMessage-able objects, it can get send to the main thread. The `Page` object calls the drawing function of the CanvasGraphics object and passes the IRQueue along. The CanvasGraphics then executes one IR command after the other.

## Objects that need IR forms

Here is a list of all the 'objects' that are more complex, depend on other objects or make other reference or are not just simple postMessage-able, such that a transformation in an IR representation is required:

* fonts
* JpegStream
* paintFormXObject
* paintImageXObject
* color spaces
* fillings/fillings
* pdf functions

The problem is that not all data is computed before a command like `paintXObject` (which might call `paintImageXObject` or `paintFormXObject`) is executed. Some of the data is computed/read from the PDF data during executing. However, as this new implementation requires the data processing and data rendering to be isolated (as there is no real chance to call back to data on the worker) this kind of data processing has to move into the PartialEvaluator's evaluate/create IRQueue step. This means some code refactoring.

Some of the objects gonna have a new `getIR()` function, that returns the postMessage-able IR representation of the object. Some also gonna have a `fromIRQueue()` function to rebuild the object from the IR representation created by the `getIR()` function.

As the object are not stored on the worker, it doesn't make sense to create a new JavaScript object just call `getIR()` on them and then wait for the GC to collect it afterwards. In places it makes sense, the now objects are replaced by simple function calls, that return the IR form directly. A new kind of object is then created, that constructs itself from the IR form such that this new object is saveable on the main thread.

E.g. the `JpegStream` gets converted to a function returning only the IR form as it is called. To store the created DOMImage and some other states on the main thread, a new `JpegStreamIR` prototype is added. A JpegObject on the main thread is then created by calling

image = new JpegStreamIR(JpegStreamIRFormAsFromWorkerSent);

Same goes for the other objects listed above.

## Implementation plan

The birds-eye thing is already implemented in the new iteration that means the basic structure how the main and worker thread talk to each other is there.

The next step is to create the IR forms for the objects listed above and transform the commands depending on them.

## Useful?

Sorry, this turned into quite some brain dump. Do you think this sounds good?