Zeppelin is a web-based notebook for data engineers that enables data-driven, interactive data analytics with Spark, Scala, and more.  The project recently reached version 0.9.0-preview2 and is being actively developed, but there are still many things to be implemented. One such thing is an API for getting comprehensive information about what's going on inside the notebook. There is already an API that completely solves the problems of high-level notebook management, but it doesn’t help if you want to do anything more complex. That was a real problem for Big Data Tools, a plugin for IntelliJ IDEA that provides integration with Spark, Hadoop, and makes it possible to edit and run notebooks in Zeppelin. The high-level API for creating and removing notebooks is not enough for the IDE. You need much lower-level access to do things like smart autocompletion or visual debugging. This is why ZTools came to exist. It’s a tool that allows you to pull information from Zeppelin, even if it's not present in the API. The server-side code is publicly available on GitHub under the Apache License 2.0. About 90% of the code is written in Scala, and the rest is in Java. ZTools will actually work with any Scala REPL, so it is not just for Zeppelin, but for dealing with Scala in general. To understand what ZTools can do, we'll look at the Variables View panel and code editor in the Big Data Tools plugin. This plugin is a part of IntelliJ Ultimate Edition, and of course, its code is not open. Nevertheless, this example shows how it might look like in your own product. When you add a new notebook to Big Data Tools, there's an option to enable ZTools integration. Now you can observe the values of local variables:  Wait, what? How is it possible to view the variables after the execution? This simple feature is an alternative to the debugger and it’ll save you a lot of time. And if that's not enough, you can now autocomplete columns of the sql.DataFrame in the editor. 
 It looks and feels like magic, and this is what ZTools provides us. At the time of the paragraph's execution, it analyzes the context on the server and sends it to the client. Let's try to understand how Zeppelin finds a list of local variables and columns. There are two ways. We can go from the server side (the open-source code for this is available on GitHub), or from the client side (a closed-source part of IntelliJ IDEA). The latter method is easier than it sounds: you can intercept traffic and see requests and responses.
Let's start with this and see how deep the rabbit hole goes. It might also teach us about an exciting hack on how to write clients for Zeppelin (spoiler: it does). You can filter traffic in different ways, but I like to use a combination of mitmproxy and Wireshark. Sorry, I don’t have the time to go into the technical details of setting up these tools in this article. Let's run some paragraphs from the standard Zeppelin example, "Spark Basic Features". Let's go to mitmproxy and see what a typical Zeppelin server response to its standard API /api/notebook looks like (see this gist for the full response). 
 There is no interesting data here, as expected. But it does lead to some interesting questions: Where are our local variables? Could it be that they are transmitted via WebSocket? Now let’s go into Wireshark and look closely at the traffic: 
 After a little digging in the traffic, we see that immediately after starting a paragraph, some JSON with the full text of the paragraph is sent to the server. This is expected. Now if you don’t blink, you’ll notice something really weird.  When you run a paragraph in Big Data Tools, for a split second an additional paragraph appears in your notebook in the IntelliJ IDEA interface, and then disappears: 
 Let's take a look at how this affects traffic. First, this temporary paragraph was sent to the server. The full text of the request is here, but we are only interested in this piece of it: This is the code of that temporary paragraph that mysteriously appeared and then disappeared. This is where the call to ZTools comes in, and it gives us a clue as to which functions we need to examine first in the GitHub repository. The server returns a lot of data chunks, but we’re only interested in two of them: Now we’ve found everything we were looking for: client requests, server response, and how exactly the additional data is transferred. Keep reading to find out whether we can apply the same technique to our own application (another spoiler: we can). You can find ZTools sources on GitHub. The repository consists mainly of two large parts: "scala-repl" and "spark".  Let’s take a look at that temporary paragraph code again: The last line lists columns in the dataframe using the Spark function Catalog.listTables() and Catalog.listColumns() then transforms them into JSON and sends them to the client. Even more interesting is where the local variables come from. When Tools is initialized, something called VariablesView is created, and there's a whole separate subproject dedicated to it. Let's take a look inside the VariablesView subproject. If you look at the tests, it should become instantly clear how to use it: It lets us interpret arbitrary Scala code and retrieve information about the variables in it. The actual interpretation of the code is carried out by the standard Scala compiler, or to be more precise, by the scala.tools.nsc.interpreter package. First, the ILoop is created, this is the main loop of the interpreter. Inside it is IMain – this is the interpreter itself. Now we call interpret(code) and that's it. Information about variables is obtained through the following trait: Of all these methods, the most important is the one that gets the list of variables. There’s nothing supernatural going on here at all, it's implemented by a direct request to the Scala interpreter, iMain: The function that renders the JSON of the "temporary paragraph" is much more interesting. Inside toJsonObject() is some complex dark magic that allows you to wander through the links between dependent variables and limit the number of collections and strings that are sent. By default, the maximum collection size is 100, and the maximum string length is 400 characters. So you don't have to worry that a terabyte of commas will be sent over the network to the client (the Big Data Tools plugin or your own app), and make everything hang forever. These limits are currently hardcoded, but if I had my way, I would move them to the settings. If you use ZTools through the Tools class, some variables are unavailable to you. Things like $intp, sc, spark, sqlContext, z, and engine are banned. If you are writing your own launcher, you should also consider creating a similar filter. Let's take advantage of the fact that ZTools can work without IntelliJ IDEA and write our own client app. A working example can be found in this GitHub repository. To prove it doesn't depend on Java, let's write everything in JavaScript (TypeScript, to be more precise) and Node.js. We will use Axios as a library for HTTP requests – as it has a pretty nice interface with promises. First, let's find a specific notebook on the server. I really like the "Spark Basic Features" notebook from the sample library. Now let's create our temporary paragraph: And new let’s create the temporary paragraph create on the server side: And run it: Then we will delete it, because it's useless now: The received response needs to be cut into two parts and parsed as JSON: Now, let's list all the variables: And here is our schema: See how easy that was? You can write this code faster than I can finish my third cup of morning coffee. It's not enough just to write a client for Zeppelin – you also need to change the Zeppelin server settings to install ZTools. But on a big production server, you probably don't have the permissions you need to do this. You'll have to beg ask the system administrators. There's absolutely no way to avoid this, and that makes me sad. As for the installation process itself, if you use vanilla ZTools without any patches, you can use our public repository. You can add it as a Maven artifact or a separate JAR file. If you want to extend ZTools yourself, you can build a project from sources using Gradle. You will get a JAR file, and then you’re free to deploy it as you like under the open-source Apache License 2.0. Big Data Tools automatically installs ZTools on the connected Zeppelin servers. We strongly recommend that you take this route. Деврел Axiom JDK, продюсер в Failover Bar