A gentle introduction to Apache Arrow with Apache Spark and Pandas | by Antonio Cachuan | Towards Data Science

This time I am going to try to explain how can we use Apache Arrow in conjunction with Apache Spark and Python. First, let me share some basic concepts about this open source project.
Apache Arrow is a cross-language development platform for in-memory data. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware. [Apache Arrow page]
In simple words, It facilitates communication between many components, for example, reading a parquet file with Python (pandas) and transforming to a Spark dataframe, Falcon Data Visualization or Cassandra without worrying about conversion.
A good question is to ask how does the data look like in memory? Well, Apache Arrow takes advantage of a columnar buffer to reduce IO and accelerate analytical processing performance.
In our case, we will use the pyarrow library to execute some basic codes and check some features. In order to install, we have two options using conda or pip commands*.
conda install -c conda-forge pyarrowpip install pyarrow
*It’s recommended to use conda in a Python 3 environment.
Apache Arrow with Pandas (Local File System)
Converting Pandas Dataframe to Apache Arrow Table
import numpy as npimport pandas as pdimport pyarrow as padf = pd.DataFrame({'one': [20, np.nan, 2.5],'two': ['january', 'february', 'march'],'three': [True, False, True]},index=list('abc'))table = pa.Table.from_pandas(df)
Pyarrow Table to Pandas Data Frame
df_new = table.to_pandas()
Read CSV
from pyarrow import csvfn = ‘data/demo.csv’table = csv.read_csv(fn)df = table.to_pandas()
Writing a parquet file from Apache Arrow
import pyarrow.parquet as pqpq.write_table(table, 'example.parquet')
Reading a parquet file
table2 = pq.read_table(‘example.parquet’)table2
Reading some columns from a parquet file
table2 = pq.read_table('example.parquet', columns=['one', 'three'])
Reading from Partitioned Datasets
dataset = pq.ParquetDataset(‘dataset_name_directory/’)table = dataset.read()table
Transforming Parquet file into a Pandas DataFrame
pdf = pq.read_pandas('example.parquet', columns=['two']).to_pandas()pdf
Avoiding pandas index
table = pa.Table.from_pandas(df, preserve_index=False)pq.write_table(table, 'example_noindex.parquet')t = pq.read_table('example_noindex.parquet')t.to_pandas()
Check metadata
parquet_file = pq.ParquetFile(‘example.parquet’)parquet_file.metadata
See data schema
parquet_file.schema
Timestamp
Remember Pandas use nanoseconds so you can truncate in milliseconds for compatibility.
pq.write_table(table, where, coerce_timestamps='ms')pq.write_table(table, where, coerce_timestamps='ms', allow_truncated_timestamps=True)
Compression
By default, Apache arrow uses snappy compression (not so compressed but easier access), although other codecs are allowed.
pq.write_table(table, where, compression='snappy')pq.write_table(table, where, compression='gzip')pq.write_table(table, where, compression='brotli')pq.write_table(table, where, compression='none')
Also, It’s possible to use more than one compression in a table
pq.write_table(table, ‘example_diffcompr.parquet’, compression={b’one’: ‘snappy’, b’two’: ‘gzip’})
Write a partitioned Parquet table
df = pd.DataFrame({‘one’: [1, 2.5, 3],                   ‘two’: [‘Peru’, ‘Brasil’, ‘Canada’],                   ‘three’: [True, False, True]},                   index=list(‘abc’))table = pa.Table.from_pandas(df)pq.write_to_dataset(table, root_path=’dataset_name’,partition_cols=[‘one’, ‘two’])
Compatibility note: if you are using pq.write_to_dataset to create a table that will then be used by HIVE then partition column values must be compatible with the allowed character set of the HIVE version you are running.
Apache Arrow with HDFS (Remote file-system)
Apache Arrow comes with bindings to a C++-based interface to the Hadoop File System. It means that we can read or download all files from HDFS and interpret directly with Python.
Connection
Host is the namenode, port is usually RPC or WEBHDFS more parameters like user, kerberos ticket are allow. It’s strongly recommended to read about the environmental variables needed.
import pyarrow as pahost = '1970.x.x.x'port = 8022fs = pa.hdfs.connect(host, port)
Optional if your connection is made front a data or edge node is possible to use just
fs = pa.hdfs.connect()
Write Parquet files to HDFS
pq.write_to_dataset(table, root_path=’dataset_name’, partition_cols=[‘one’, ‘two’], filesystem=fs)
Read CSV from HDFS
import pandas as pdfrom pyarrow import csvimport pyarrow as pafs = pa.hdfs.connect()with fs.open(‘iris.csv’, ‘rb’) as f: df = pd.read_csv(f, nrows = 10)df.head()
Read Parquet File from HDFS
There is two forms to read a parquet file from HDFS
Using pandas and Pyarrow engine
import pandas as pdpdIris = pd.read_parquet(‘hdfs:///iris/part-00000–27c8e2d3-fcc9–47ff-8fd1–6ef0b079f30e-c000.snappy.parquet’, engine=’pyarrow’)pdTrain.head()
Pyarrow.parquet
import pyarrow.parquet as pqpath = ‘hdfs:///iris/part-00000–71c8h2d3-fcc9–47ff-8fd1–6ef0b079f30e-c000.snappy.parquet’table = pq.read_table(path)table.schemadf = table.to_pandas()df.head()
Other files extensions
As we can store any kind of files (SAS, STATA, Excel, JSON or objects), the majority of then are easily interpreted by Python. To accomplish that we’ll use the open function that returns a buffer object that many pandas function like read_sas, read_json could receive as input instead of a string URL.
SAS
import pandas as pdimport pyarrow as pafs = pa.hdfs.connect()with fs.open(‘/datalake/airplane.sas7bdat’, ‘rb’) as f: sas_df = pd.read_sas(f, format='sas7bdat')sas_df.head()
Excel
import pandas as pdimport pyarrow as pafs = pa.hdfs.connect()with fs.open(‘/datalake/airplane.xlsx’, ‘rb’) as f: g.download('airplane.xlsx')ex_df = pd.read_excel('airplane.xlsx')
JSON
import pandas as pdimport pyarrow as pafs = pa.hdfs.connect()with fs.open(‘/datalake/airplane.json’, ‘rb’) as f: g.download('airplane.json')js_df = pd.read_json('airplane.json')
Download files from HDFS
If we just need to download the file, Pyarrow provides us with the download function to save the file in local.
import pandas as pdimport pyarrow as pafs = pa.hdfs.connect()with fs.open(‘/datalake/airplane.cs’, ‘rb’) as f: g.download('airplane.cs')
Upload files to HDFS
If we just need to download the file, Pyarrow provides us with the download function to save the file in local.
import pyarrow as pafs = pa.hdfs.connect()with open(‘settings.xml’) as f: pa.hdfs.HadoopFileSystem.upload(fs, ‘/datalake/settings.xml’, f)
Apache Arrow with Apache Spark
Apache Arrow is integrated with Spark since version 2.3, exists good presentations about optimizing times avoiding serialization & deserialization process and integrating with other libraries like a presentation about accelerating Tensorflow Apache Arrow on Spark from Holden Karau.
Exist other useful articles like one published by Brian Cutler and really good examples in the Spark’s official documentation
Some interesting uses of Apache Arrow are:
Speeding upconversion from Pandas Data Frame to Spark Data Frame
Speeding upconversion from Spark Data Frame to Pandas Data Frame
Using with Pandas UDF (a.k.a Vectorized UDFs)
Optimizing R with Apache Spark
The third item will be part from a next article since It’s a very interesting topic in order to expand the integration between Pandas and Spark without losing performance, for the fourth item I recommend you to read the article (was published in 2019!) to get know more about it.
Let’s test the conversion among Pandas and Spark first without modifying anything and then allowing Arrow.
from pyspark.sql import SparkSessionwarehouseLocation = “/antonio”spark = SparkSession\.builder.appName(“demoMedium”)\.config(“spark.sql.warehouse.dir”, warehouseLocation)\.enableHiveSupport()\.getOrCreate()#Create test Spark DataFramefrom pyspark.sql.functions import randdf = spark.range(1 << 22).toDF(“id”).withColumn(“x”, rand())df.printSchema()#Benchmark time%time pdf = df.toPandas()spark.conf.set(“spark.sql.execution.arrow.enabled”, “true”)%time pdf = df.toPandas()pdf.describe()
In the results is clearly more convenient to use Arrow to decrease time conversion.
If we need to test the contrary case (pandas to spark df) also we see an optimization in time.
%time df = spark.createDataFrame(pdf)spark.conf.set("spark.sql.execution.arrow.enabled", "false")%time df = spark.createDataFrame(pdf)df.describe().show()
In conclusion
This article's intention was to discover and understand about Apache Arrow and how it works with Apache Spark and Pandas, also I suggest you check the official page of It to know more about other possible integration like CUDA or C++, also if you want to go deeper and learn more about Apache Spark, I think Spark: The Definitive Guide is an excellent book.
PS if you have any questions, or would like something clarified, you can find me on Twitter and LinkedIn. I recently published A gentle introduction to Apache Druid a new Apache project ideal for analyzing billions of rows.