Apache Beam Python入門

# 1. Preparation

## 1.1. Python version

```
python -V
Python 3.9.9
```

## 1.2. Install Dependencies

```
pip install apache-beam
```

## 1.3. Prepare input.txt

```input.txt
This guide shows you how to set up your Python development environment, get the Apache Beam SDK for Python, and run an example pipeline.
```

# 2. Run

```
python -m apache_beam.examples.wordcount --input input.txt --output output.txt
```

<details>

```
INFO:root:Missing pipeline option (runner). Executing pipeline using the default runner: DirectRunner.
INFO:root:Default Python SDK image for environment is apache/beam_python3.9_sdk:2.41.0
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function annotate_downstream_side_inputs at 0x116968b80> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function fix_side_input_pcoll_coders at 0x116968ca0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function pack_combiners at 0x11696a1f0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function lift_combiners at 0x11696a280> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function expand_sdf at 0x11696a430> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function expand_gbk at 0x11696a4c0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sink_flattens at 0x11696a5e0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function greedily_fuse at 0x11696a670> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function read_to_impulse at 0x11696a700> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function impulse_to_input at 0x11696a790> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function sort_stages at 0x11696a9d0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function add_impulse_to_dangling_transforms at 0x11696aaf0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function setup_timer_mapping at 0x11696a940> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:==================== <function populate_data_channel_coders at 0x11696aa60> ====================
INFO:apache_beam.runners.worker.statecache:Creating state cache with size 100
INFO:apache_beam.runners.portability.fn_api_runner.worker_handlers:Created Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler object at 0x130b64b80> for environment ref_Environment_default_environment_1 (beam:env:embedded_python:v1, b'')
INFO:apache_beam.io.filebasedsink:Starting finalize_write threads with num_shards: 1 (skipped: 0), batches: 1, num_threads: 1
INFO:apache_beam.io.filebasedsink:Renamed 1 shards in 0.01 seconds.
```

</details>

# 3. Check the output

```
cat output.txt-00000-of-00001 
This: 1
guide: 1
shows: 1
you: 1
how: 1
to: 1
set: 1
up: 1
your: 1
Python: 2
development: 1
environment: 1
get: 1
the: 1
Apache: 1
Beam: 1
SDK: 1
for: 1
and: 1
run: 1
an: 1
example: 1
pipeline: 1
```

# References

1. https://beam.apache.org/get-started/quickstart-py/