Project: Beam (App Definition and Development / Streaming & Messaging)
File: differences-from-pandas.md

type: languages title: "Differences from pandas" <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> The Apache Beam DataFrame API aims to be a drop-in replacement for pandas, but there are a few differences to be aware of. This page describes divergences between the Beam and pandas APIs and provides tips for working with the Beam DataFrame API. See the for a full reference for which operations and arguments are supported in the Beam DataFrame API. Beam operations are always associated with a pipeline. To read source data into a Beam DataFrame, you have to apply the source to a pipeline object. For example, to read input from a CSV file, you could use as follows: df = p | beam.dataframe.io.read_csv(...) This is similar to pandas , but `df` is a deferred Beam DataFrame representing the contents of the file. The input filename can be any file pattern understood by . For an example of using sources and sinks with the DataFrame API, see . The sections below describe classes of operations that are not yet supported, or supported with caveats, by the Beam DataFrame API. Workarounds are suggested where applicable. Examples: , To support distributed processing, Beam invokes DataFrame operations on subsets of data in parallel. Some DataFrame operations cant be parallelized, and these operations raise a error by default. Workaround If you want to use a non-parallelizable operation, you can guard it with a `beam.dataframe.allownonparallel_operations` block. For example: from apache_beam import dataframe with dataframe.allownonparallel_operations(): quantiles = df.quantile() Note that this collects the entire input dataset on a single node, so theres a risk of running out of memory. You should only use this workaround if youre sure that the input is small enough to process on a single worker. Examples: , , Beam DataFrame operations are deferred, but the schemas of the resulting DataFrames are not, meaning that result columns must be computable without access to the data. Some DataFrame operations cant support this usage, so they cant be implemented. These operations raise a . <!-- TODO(https://github.com/apache/beam/issues/20958): Document the use of categorical columns as a workaround --> Currently theres no workaround for this issue. But in the future, Beam Dataframe may support non-deferred column operations on categorical

This work is being tracked in . Examples: , , Its infeasible to implement DataFrame operations that produce non-deferred values or plots because Beam is a deferred API. If these operations are invoked, they will raise a . These operations may be supported in the future through a tighter integration with Interactive Beam. To track progress on this issue, follow . If you think we should prioritize this work you can also [contact us](/community/contact-us/) to let us know. Workaround If youre using , you can use `collect` to bring a dataset into local memory and then perform these operations. Examples: , , Beam PCollections are inherently unordered, so pandas operations that are sensitive to the ordering of rows are not supported. These operations raise a . Order-sensitive operations may be supported in the future. To track progress on this issue, follow . If you think we should prioritize this work you can also to let us know. Workaround If youre using , you can use `collect` to bring a dataset into local memory and then perform these operations. Alternatively, there may be ways to rewrite your code so that its not order sensitive. For example, pandas users often call the order-sensitive operation to peek at data, but if you just want to view a subset of elements, you can also use , which doesnt require you to collect the data first. Similarly, you could use `nlargest` instead of `sort_values(...).`. Some DataFrame operations produce deferred scalars. In Beam, actual computation of the values is deferred, and so the values are not available for control flow. For example, you can compute a sum with `Series.sum`, but you cant immediately branch on the result, because the result data is not immediately available. `Series.is_unique` is a similar example. Using a deferred scalar for branching logic or truth tests raises a . The Beam DataFrame API implements many of the commonly used pandas DataFrame operations, and were actively working to support the remaining operations. But pandas has a large API, and there are still gaps (). If you invoke an operation that hasnt been implemented yet, it will raise a `NotImplementedError`. Please if you encounter a missing operation that you think should be prioritized. Interactive Beam is a module designed for use in interactive notebooks. The module, which by convention is imported as `ib`, provides an `ib.collect` function that brings a `PCollection` or deferred DataFrame into local memory as a pandas DataFrame. After using `ib.collect` to materialize a deferred DataFrame you will be able to perform any operation in the pandas API, not just those that are supported in Beam. {{< button-colab url="https://colab.research.google.com/github/apache/beam/blob/master/examples/notebooks/tour-of-beam/dataframes.ipynb" >}} To get started with Beam in a notebook, see .