How To Visualize Spark DataFrames In Scala | by Chengzhi Zhao | Towards Data Science

Visualization of a dataset is a compelling way to explore data and delivers meaningful information to the end-users. With Spark DataFrame, data processing on a large scale has never been more natural than current stacks. For people who write code in Python, there are many visualization options to choose; data visualization may not be a concern with PySpark engineers. However, for people writing Spark in Scala, there are not numerous open-source options available.
For Spark In Scala DataFrame visualization, if you search “Spark In Scala DataFrame Visualization” on Google, a list of options ties strictly to vendors or commercial solutions. If you are using Databricks, the functiondisplay is handy. If you are using HDInsight Spark, a build-in visualization is available. If you are using Zeppelin (open-source), the visualization button is possible to make it easy. However, if you don’t have any of the environment mentioned above, and you still want to use open-source like Jupyter Notebook, data visualization is not a mission impossible here. Fortunately, there are customized functions, and libraries can make this process simple. In this article, we are going to explore a better visualization experience for ONLY Scala.
In Spark, a simple visualization in the console is the show function. The show function displays a few records (default is 20 rows) from DataFrame into a tabular form. The default behavior of the show function is truncate enabled, which won’t display a value if it’s longer than 20 characters. In this way, you might have everything display about right. Sometimes you may want to disable the truncate to view more content in a cell. Then your data showed probably would be messy as it won’t line up, and it becomes tough to read.
The following example we have a column called extremely_long_str , which we set it on purpose to observe the behavior of the extended content within a cell. In this case, the show function won’t format nicely.
import spark.implicits._val columns = Seq(“language”,”users_count”, “extremely_long_str”)val data = Seq((“Java”, “20000”,”Short Text”), (“Python”, “100000”,”Medium Text, Medium Text, Medium Text”), (“Scala”, “3000”,”Extremely Long Text, Extremely Long Text, Extremely Long Text, Extremely Long Text, Extremely Long Text,Extremely Long Text,Extremely Long Text, Extremely Long Text, Extremely Long Text, Extremely Long Text, Extremely Long Text, Extremely Long Text,Extremely Long Text,Extremely Long Text”))val rdd = spark.sparkContext.parallelize(data)val df = spark.createDataFrame(rdd).toDF(columns:_*)df.show(false)
Once you executed the following code, it displays the following lines. We could recognize that one extra-long record which doesn’t fit into one row. If you have several hundreds of lines, it becomes difficult to read since the context within a cell breaks into multiple lines. As the turncate is off, the long context breaks the well-formatted show function. You may notice it becomes disturbing to read, and it is even more troublesome if you have multiple columns layout like this.
In Jupyter notebook, to fix the alignment issue. We can apply HTML to display the content instead of using the show function. HTML would be much flexible here, and it can manage the cells merging so it would display more beautiful in multiple lines, and the output here is more comfortable to read.
To get this work, all you need is to install a Jupyter Notebook kernel, which is call Almond (A Scala kernel for Jupyter), and implement a customized function. To install the Almond kernel in Jupyter Notebook, you can follow the instruction.
Aivean posted a useful function on Github for this, and once you add the helper function, you can calldf.showHTML(10, 300) function, which generated an HTML code block wrap with the DataFrame result, and displays ten rows with 300 characters per cell. It looks much better now in Jupyter Notebook as the image shown above. The function to add looks like the following:
// Credit to Aiveanimplicit class RichDF(val ds:DataFrame) {    def showHTML(limit:Int = 20, truncate: Int = 20) = {        import xml.Utility.escape        val data = ds.take(limit)        val header = ds.schema.fieldNames.toSeq                val rows: Seq[Seq[String]] = data.map { row =>          row.toSeq.map { cell =>            val str = cell match {              case null => "null"              case binary: Array[Byte] => binary.map("%02X".format(_)).mkString("[", " ", "]")              case array: Array[_] => array.mkString("[", ", ", "]")              case seq: Seq[_] => seq.mkString("[", ", ", "]")              case _ => cell.toString            }            if (truncate > 0 && str.length > truncate) {              // do not show ellipses for strings shorter than 4 characters.              if (truncate < 4) str.substring(0, truncate)              else str.substring(0, truncate - 3) + "..."            } else {              str            }          }: Seq[String]        }publish.html(s""" <table>                <tr>                 ${header.map(h => s"<th>${escape(h)}</th>").mkString}                </tr>                ${rows.map { row =>                  s"<tr>${row.map{c => s"<td>${escape(c)}</td>" }.mkString}</tr>"                }.mkString}            </table>        """)            }}
Vegas is a Scala API for declarative, statistical data visualizations. It integrated well with Scala as well as the modern data framework such as Apache Spark and Apache Flink. Since Vegas is declarative, all we need to do is define data sources and pass arguments on how to display the plots without explicitly write down more extra codes.
First, you’d need to add the following two dependencies
import $ivy.`org.vegas-viz:vegas_2.11:0.3.11`import $ivy.`org.vegas-viz:vegas-spark_2.11:0.3.11`
Since we have a Spark DataFrame we have defined earlier, we can reuse it. Once you have the DataFrame defined, the rest is to point withDataFrame to the Spark DataFrame, so Vegas knows how to parse the Spark DataFrame as your data source. To present a chart beautifully, you may want to sort the x-axis, otherwise the plot sorts and displays by language name, which is the default behavior.
import vegas._import vegas.render.WindowRenderer._import vegas.sparkExt._Vegas(“Language Usage”) .withDataFrame(df) .mark(Bar) // Change to .mark(Area) .encodeX(“language”, Nom, sortField=Sort(“users_count”, AggOps.Mean)) .encodeY(“users_count”, Quant) .show
Vegas is an extraordinary library to use, and it works seamlessly with Scala and Spark. Although the plot in Vegas looks cool, you might not only limit yourself to only one visualization option. Also, you may want to have a more interactive mode with the chart. Plotly might be the right choice here. Plotly is another remarkable data visualization framework, and it gains popularity in Python and JavaScript already. To get Plotly work with Scala and Spark, we’d need to reshape our data more due to Plotly currently doesn’t support Spark DataFrame directly.
First, you’d need to install plotly-scala for Jupyter lab
jupyter labextension install @jupyterlab/plotly-extension
Then you’d need to change DataFrame to RDD and collect to force data collection to the driver node. To avoid receiving too much data to the driver, before collecting data on Spark driver, you’d need to filter or aggregated your dataset close to the final result and don’t rely on visualization framework to perform data transformations.
val (x, y) = df.collect.map(r=>(r(0).toString, r(1).toString.toInt)).toList.unzipBar(x, y).plot()
A more refined feature in Plotly is its charts are more interactive than the ones created by Vegas. You can hover on the bar chart and see the value of the data, or choose options on the top right like zoom in/out to fit your requirements.
Although there are a few data visualization options in Scala, it is still possible to build impressive and creative charts to communicate information via data. For people who write code in Scala for Spark, with additional transformations, we can still leverage some open-source libraries to visualize data in Scala. I hope this article can introduce some ideas on how to visualize Spark DataFrame in Scala to help you get a better visualization experience for Scala.