Repository: apache/beam
Pull request #34569: Add documentation for Beam YAML testing.
Author: login: robertwb
Labels: name: python
name: website
name: Next Action: Reviewers
name: yaml
Created: 2025-04-07T21:57:49Z | Merged: 2025-04-10T01:40:56Z


------------------------

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

 - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
 - [ ] Update `CHANGES.md` with noteworthy changes.
 - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).

To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)

GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[![Build python source distribution and wheels](https://github.com/apache/beam/actions/workflows/build_wheels.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[![Python tests](https://github.com/apache/beam/actions/workflows/python_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[![Java tests](https://github.com/apache/beam/actions/workflows/java_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[![Go tests](https://github.com/apache/beam/actions/workflows/go_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)

See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.



Diff:
diff --git a/sdks/python/apache_beam/yaml/main.py b/sdks/python/apache_beam/yaml/main.py
index 26b8512cb656..35879514ce15 100644
--- a/sdks/python/apache_beam/yaml/main.py
+++ b/sdks/python/apache_beam/yaml/main.py
@@ -18,6 +18,9 @@
 import argparse
 import contextlib
 import json
+import os
+import sys
+import unittest
 
 import yaml
 
@@ -26,7 +29,9 @@
 from apache_beam.transforms import resources
 from apache_beam.typehints.schemas import LogicalType
 from apache_beam.typehints.schemas import MillisInstant
+from apache_beam.yaml import yaml_testing
 from apache_beam.yaml import yaml_transform
+from apache_beam.yaml import yaml_utils
 
 
 def _preparse_jinja_flags(argv):
@@ -90,6 +95,28 @@ def _parse_arguments(argv):
       type=json.loads,
       help='A json dict of variables used when invoking the jinja preprocessor '
       'on the provided yaml pipeline.')
+  parser.add_argument(
+      '--test',
+      action=argparse.BooleanOptionalAction,
+      help='Run the tests associated with the given pipeline, rather than the '
+      'pipeline itself.')
+  parser.add_argument(
+      '--fix_tests',
+      action=argparse.BooleanOptionalAction,
+      help='Update failing test expectations to match the actual ouput. '
+      'Requires --test_suite if the pipeline uses jinja formatting.')
+  parser.add_argument(
+      '--create_test',
+      action=argparse.BooleanOptionalAction,
+      help='Automatically creates a regression test for the given pipeline, '
+      'adding it to the pipeline spec or test suite dependon on whether '
+      '--test_suite is given. '
+      'Requires --test_suite if the pipeline uses jinja formatting.')
+  parser.add_argument(
+      '--test_suite',
+      help='Run the given tests against the given pipeline, rather than the '
+      'pipeline itself. '
+      'Should be a file containing a list of yaml test specifications.')
   return parser.parse_known_args(argv)
 
 
@@ -130,12 +157,109 @@ def run(argv=None):
       print('Running pipeline...')
 
 
-def build_pipeline_components_from_argv(argv):
+def run_tests(argv=None, exit=True):
+  known_args, pipeline_args, _, pipeline_yaml = _build_pipeline_yaml_from_argv(
+      argv)
+  pipeline_spec = yaml.load(pipeline_yaml, Loader=yaml_transform.SafeLineLoader)
+  options = _build_pipeline_options(pipeline_spec, pipeline_args)
+
+  if known_args.create_test and known_args.fix_tests:
+    raise ValueError(
+        'At most one of --create_test and --fix_tests may be specified.')
+  elif known_args.create_test:
+    result = unittest.TestResult()
+    tests = []
+  else:
+    if known_args.test_suite:
+      with open(known_args.test_suite) as fin:
+        test_suite_holder = yaml.load(
+            fin, Loader=yaml_transform.SafeLineLoader) or {}
+    else:
+      test_suite_holder = pipeline_spec
+    test_specs = test_suite_holder.get('tests', [])
+    if not isinstance(test_specs, list):
+      raise TypeError('tests attribute must be a list of test specifications.')
+    elif not test_specs:
+      raise RuntimeError(
+          'No tests found. '
+          "If you haven't added a set of tests yet, you can get started by "
+          'running your pipeline with the --create_test flag enabled.')
+
+    with _fix_xlang_instant_coding():
+      tests = [
+          yaml_testing.YamlTestCase(
+              pipeline_spec, test_spec, options, known_args.fix_tests)
+          for test_spec in test_specs
+      ]
+      suite = unittest.TestSuite(tests)
+      result = unittest.TextTestRunner().run(suite)
+
+  if known_args.fix_tests or known_args.create_test:
+    update_tests(known_args, pipeline_yaml, pipeline_spec, options, tests)
+
+  if exit:
+    # emulates unittest.main()
+    sys.exit(0 if result.wasSuccessful() else 1)
+  else:
+    if not result.wasSuccessful():
+      raise RuntimeError(result)
+
+
+def update_tests(known_args, pipeline_yaml, pipeline_spec, options, tests):
+  if known_args.test_suite:
+    path = known_args.test_suite
+    if not os.path.exists(path) and known_args.create_test:
+      with open(path, 'w') as fout:
+        fout.write('tests: []')
+  elif known_args.yaml_pipeline_file:
+    path = known_args.yaml_pipeline_file
+  else:
+    raise RuntimeError(
+        'Test fixing only supported for file-backed tests. '
+        'Please use the --test_suite flag.')
+  with open(path) as fin:
+    original_yaml = fin.read()
+  if path == known_args.yaml_pipeline_file and pipeline_yaml.strip(
+  ) != original_yaml.strip():
+    raise RuntimeError(
+        'In-file test fixing not yet supported for templated pipelines. '
+        'Please use the --test_suite flag.')
+  updated_spec = yaml.load(original_yaml, Loader=yaml.SafeLoader) or {}
+
+  if known_args.fix_tests:
+    updated_spec['tests'] = [test.fixed_test() for test in tests]
+
+  if known_args.create_test:
+    if 'tests' not in updated_spec:
+      updated_spec['tests'] = []
+    updated_spec['tests'].append(
+        yaml_testing.create_test(pipeline_spec, options))
+
+  updated_yaml = yaml_utils.patch_yaml(original_yaml, updated_spec)
+  with open(path, 'w') as fout:
+    fout.write(updated_yaml)
+
+
+def _build_pipeline_yaml_from_argv(argv):
   argv = _preparse_jinja_flags(argv)
   known_args, pipeline_args = _parse_arguments(argv)
   pipeline_template = _pipeline_spec_from_args(known_args)
   pipeline_yaml = yaml_transform.expand_jinja(
       pipeline_template, known_args.jinja_variables or {})
+  return known_args, pipeline_args, pipeline_template, pipeline_yaml
+
+
+def _build_pipeline_options(pipeline_spec, pipeline_args):
+  return beam.options.pipeline_options.PipelineOptions(
+      pipeline_args,
+      pickle_library='cloudpickle',
+      **yaml_transform.SafeLineLoader.strip_metadata(
+          pipeline_spec.get('options', {})))
+
+
+def build_pipeline_components_from_argv(argv):
+  (known_args, pipeline_args, pipeline_template,
+   pipeline_yaml) = _build_pipeline_yaml_from_argv(argv)
   display_data = {
       'yaml': pipeline_yaml,
       'yaml_jinja_template': pipeline_template,
@@ -154,11 +278,7 @@ def build_pipeline_components_from_yaml(
     pipeline_yaml, pipeline_args, validate_schema='generic', pipeline_path=''):
   pipeline_spec = yaml.load(pipeline_yaml, Loader=yaml_transform.SafeLineLoader)
 
-  options = beam.options.pipeline_options.PipelineOptions(
-      pipeline_args,
-      pickle_library='cloudpickle',
-      **yaml_transform.SafeLineLoader.strip_metadata(
-          pipeline_spec.get('options', {})))
+  options = _build_pipeline_options(pipeline_spec, pipeline_args)
 
   def constructor(root):
     if 'resource_hints' in pipeline_spec.get('pipeline', {}):
@@ -180,4 +300,7 @@ def constructor(root):
 if __name__ == '__main__':
   import logging
   logging.getLogger().setLevel(logging.INFO)
-  run()
+  if '--test' in sys.argv:
+    run_tests()
+  else:
+    run()
diff --git a/sdks/python/apache_beam/yaml/main_test.py b/sdks/python/apache_beam/yaml/main_test.py
index d5fbfedc0349..d233e0e2d73c 100644
--- a/sdks/python/apache_beam/yaml/main_test.py
+++ b/sdks/python/apache_beam/yaml/main_test.py
@@ -38,6 +38,44 @@
     - type: WriteToText
       config:
         path: PATH
+
+tests:
+  - name: InlineTest
+    mock_outputs:
+      - name: Create
+        elements: ['a', 'b', 'c']
+    expected_inputs:
+      - name: WriteToText
+        elements:
+          - {element: a}
+          - {element: b}
+          - {element: c}
+'''
+
+PASSING_TEST_SUITE = '''
+tests:
+  - name: ExternalTest  # comment
+    mock_outputs:
+      - name: Create
+        elements: ['a', 'b', 'c']
+    expected_inputs:
+      - name: WriteToText
+        elements:
+          - element: a
+          - element: b
+          - element: c
+'''
+
+FAILING_TEST_SUITE = '''
+tests:
+  - name: ExternalTest  # comment
+    mock_outputs:
+      - name: Create
+        elements: ['a', 'b', 'c']
+    expected_inputs:
+      - name: WriteToText
+        elements:
+          - element: x
 '''
 
 
@@ -113,6 +151,82 @@ def test_jinja_datetime(self):
         self.assertEqual(
             fin.read().strip(), datetime.datetime.now().strftime("%Y-%m-%d"))
 
+  def test_inline_test_specs(self):
+    main.run_tests(['--yaml_pipeline', TEST_PIPELINE, '--test'], exit=False)
+
+  def test_external_test_specs(self):
+    with tempfile.TemporaryDirectory() as tmpdir:
+      good_suite = os.path.join(tmpdir, 'good.yaml')
+      with open(good_suite, 'w') as fout:
+        fout.write(PASSING_TEST_SUITE)
+      bad_suite = os.path.join(tmpdir, 'bad.yaml')
+      with open(bad_suite, 'w') as fout:
+        fout.write(FAILING_TEST_SUITE)
+
+      # Must pass.
+      main.run_tests([
+          '--yaml_pipeline',
+          TEST_PIPELINE,
+          '--test_suite',
+          good_suite,
+      ],
+                     exit=False)
+
+      # Must fail. (Ensures testing is not a no-op.)
+      with self.assertRaisesRegex(Exception, 'errors=1 failures=0'):
+        main.run_tests([
+            '--yaml_pipeline',
+            TEST_PIPELINE,
+            '--test_suite',
+            bad_suite,
+        ],
+                       exit=False)
+
+  def test_fix_suite(self):
+    with tempfile.TemporaryDirectory() as tmpdir:
+      test_suite = os.path.join(tmpdir, 'tests.yaml')
+      with open(test_suite, 'w') as fout:
+        fout.write(FAILING_TEST_SUITE)
+
+      main.run_tests([
+          '--yaml_pipeline',
+          TEST_PIPELINE,
+          '--test_suite',
+          test_suite,
+          '--fix_tests'
+      ],
+                     exit=False)
+
+      with open(test_suite) as fin:
+        self.assertEqual(fin.read(), PASSING_TEST_SUITE)
+
+  def test_create_test(self):
+    with tempfile.TemporaryDirectory() as tmpdir:
+      test_suite = os.path.join(tmpdir, 'tests.yaml')
+      with open(test_suite, 'w') as fout:
+        fout.write('')
+
+      main.run_tests([
+          '--yaml_pipeline',
+          TEST_PIPELINE.replace('ELEMENT', 'x'),
+          '--test_suite',
+          test_suite,
+          '--create_test'
+      ],
+                     exit=False)
+
+      with open(test_suite) as fin:
+        self.assertEqual(
+            fin.read(),
+            '''
+tests:
+- mock_outputs: []
+  expected_inputs:
+  - name: WriteToText
+    elements:
+    - element: x
+'''.lstrip())
+
 
 if __name__ == '__main__':
   logging.getLogger().setLevel(logging.INFO)
diff --git a/sdks/python/apache_beam/yaml/readme_test.py b/sdks/python/apache_beam/yaml/readme_test.py
index fffacf7f42b9..4a9fad991c53 100644
--- a/sdks/python/apache_beam/yaml/readme_test.py
+++ b/sdks/python/apache_beam/yaml/readme_test.py
@@ -34,7 +34,9 @@
 from apache_beam.options.pipeline_options import PipelineOptions
 from apache_beam.typehints import trivial_inference
 from apache_beam.yaml import yaml_provider
+from apache_beam.yaml import yaml_testing
 from apache_beam.yaml import yaml_transform
+from apache_beam.yaml import yaml_utils
 
 
 class FakeSql(beam.PTransform):
@@ -288,6 +290,7 @@ def extract_name(input_spec):
     return input_spec.get('name', input_spec.get('type'))
 
   code_lines = None
+  last_pipeline = None
   for ix, line in enumerate(markdown_lines):
     line = line.rstrip()
     if line == '```':
@@ -320,12 +323,30 @@ def extract_name(input_spec):
             ] + ['    ' + line for line in code_lines]
           if code_lines[0] == 'pipeline:':
             yaml_pipeline = '\n'.join(code_lines)
-            if 'providers:' in yaml_pipeline:
+            last_pipeline = yaml_pipeline
+            if 'providers:' in yaml_pipeline or 'tests:' in yaml_pipeline:
               test_type = 'PARSE'
             yield test_name, create_test_method(
                 test_type,
                 test_name,
                 yaml_pipeline)
+          if 'tests:' in code_lines:
+            test_spec = '\n'.join(code_lines)
+            if code_lines[0] == 'pipeline:':
+              yaml_pipeline = '\n'.join(code_lines)
+            else:
+              yaml_pipeline = last_pipeline
+            for sub_ix, test_spec in enumerate(yaml.load(
+                '\n'.join(code_lines),
+                Loader=yaml_utils.SafeLineLoader)['tests']):
+              suffix = test_spec.get('name', str(sub_ix))
+              yield (
+                  test_name + '_' + suffix,
+                  # The yp=... ts=... is to capture the looped closure values.
+                  lambda _,
+                  yp=yaml_pipeline,
+                  ts=test_spec: yaml_testing.run_test(yp, ts))
+
         code_lines = None
     elif code_lines is not None:
       code_lines.append(line)
@@ -358,6 +379,9 @@ def createTestSuite(name, path):
 JoinTest = createTestSuite(
     'JoinTest', os.path.join(YAML_DOCS_DIR, 'yaml-join.md'))
 
+TestingTest = createTestSuite(
+    'TestingTest', os.path.join(YAML_DOCS_DIR, 'yaml-testing.md'))
+
 if __name__ == '__main__':
   parser = argparse.ArgumentParser()
   parser.add_argument('--render_dir', default=None)
diff --git a/sdks/python/apache_beam/yaml/yaml_testing.py b/sdks/python/apache_beam/yaml/yaml_testing.py
new file mode 100644
index 000000000000..689ba208a9ae
--- /dev/null
+++ b/sdks/python/apache_beam/yaml/yaml_testing.py
@@ -0,0 +1,528 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import collections
+import functools
+import json
+import random
+import unittest
+import uuid
+from typing import Dict
+from typing import List
+from typing import Mapping
+from typing import Optional
+from typing import Tuple
+from typing import TypeVar
+from typing import Union
+
+import yaml
+
+import apache_beam as beam
+from apache_beam.testing.util import assert_that
+from apache_beam.testing.util import equal_to
+from apache_beam.yaml import yaml_provider
+from apache_beam.yaml import yaml_transform
+from apache_beam.yaml import yaml_utils
+
+
+class YamlTestCase(unittest.TestCase):
+  def __init__(self, pipeline_spec, test_spec, options, fix_tests):
+    super().__init__()
+    self._pipeline_spec = pipeline_spec
+    self._test_spec = test_spec
+    self._options = options
+    self._fix_tests = fix_tests
+
+  def runTest(self):
+    self._fixes = run_test(
+        self._pipeline_spec, self._test_spec, self._options, self._fix_tests)
+
+  def fixed_test(self):
+    fixed_test_spec = yaml_transform.SafeLineLoader.strip_metadata(
+        self._test_spec)
+    if self._fixes:
+      expectation_by_id = {(loc, expectation['name']): expectation
+                           for loc in ('expected_inputs', 'expected_outputs')
+                           for expectation in fixed_test_spec.get(loc, [])}
+      for name_loc, values in self._fixes.items():
+        expectation_by_id[name_loc]['elements'] = sorted(values, key=json.dumps)
+    return fixed_test_spec
+
+  def id(self):
+    return (
+        self._test_spec.get('name', 'unknown') +
+        f' (line {yaml_transform.SafeLineLoader.get_line(self._test_spec)})')
+
+  def __str__(self):
+    return self.id()
+
+
+def run_test(pipeline_spec, test_spec, options=None, fix_failures=False):
+  if isinstance(pipeline_spec, str):
+    pipeline_spec = yaml.load(pipeline_spec, Loader=yaml_utils.SafeLineLoader)
+
+  pipeline_spec = _preprocess_for_testing(pipeline_spec)
+
+  transform_spec, recording_ids = inject_test_tranforms(
+      pipeline_spec,
+      test_spec,
+      fix_failures)
+
+  allowed_sources = set(test_spec.get('allowed_sources', []) + ['Create'])
+  for transform in transform_spec['transforms']:
+    name_or_type = transform.get('name', transform['type'])
+    if (not yaml_transform.empty_if_explicitly_empty(transform.get('input', []))
+        and not transform.get('name') in allowed_sources and
+        not transform['type'] in allowed_sources):
+      raise ValueError(
+          f'Non-mocked source {name_or_type} '
+          f'at line {yaml_transform.SafeLineLoader.get_line(transform)}')
+
+  if options is None:
+    options = beam.options.pipeline_options.PipelineOptions(
+        pickle_library='cloudpickle',
+        **yaml_transform.SafeLineLoader.strip_metadata(
+            pipeline_spec.get('options', {})))
+
+  with beam.Pipeline(options=options) as p:
+    _ = p | yaml_transform.YamlTransform(
+        transform_spec,
+        providers={'AssertEqualAndRecord': AssertEqualAndRecord})
+
+  if fix_failures:
+    fixes = {}
+    for recording_id in recording_ids:
+      if AssertEqualAndRecord.has_recorded_result(recording_id):
+        fixes[recording_id[1:]] = [
+            _try_row_as_dict(row)
+            for row in AssertEqualAndRecord.get_recorded_result(recording_id)
+        ]
+        AssertEqualAndRecord.remove_recorded_result(recording_id)
+    return fixes
+
+
+def _preprocess_for_testing(pipeline_spec):
+  spec = yaml_transform.pipeline_as_composite(pipeline_spec['pipeline'])
+  # These are idempotent, so it's OK to do them preemptively.
+  for phase in [
+      yaml_transform.ensure_transforms_have_types,
+      yaml_transform.preprocess_source_sink,
+      yaml_transform.preprocess_chain,
+      yaml_transform.tag_explicit_inputs,
+      yaml_transform.normalize_inputs_outputs,
+  ]:
+    spec = yaml_transform.apply_phase(phase, spec)
+
+  return spec
+
+
+def validate_test_spec(test_spec):
+  if not isinstance(test_spec, dict):
+    raise TypeError(
+        f'Test specification must be an object, got {type(test_spec)}')
+  identifier = (
+      test_spec.get('name', 'unknown') +
+      f' at line {yaml_transform.SafeLineLoader.get_line(test_spec)}')
+
+  if not isinstance(test_spec.get('allowed_sources', []), list):
+    raise TypeError(
+        f'allowed_sources of test specification {identifier} '
+        f'must be a list, got {type(test_spec["allowed_sources"])}')
+
+  if (not test_spec.get('expected_outputs', []) and
+      not test_spec.get('expected_inputs', [])):
+    raise ValueError(
+        f'test specification {identifier} '
+        f'must have at least one expected_outputs or expected_inputs')
+
+  unknown_attrs = set(
+      yaml_transform.SafeLineLoader.strip_metadata(test_spec).keys()) - set([
+          'name',
+          'mock_inputs',
+          'mock_outputs',
+          'expected_outputs',
+          'expected_inputs',
+          'allowed_sources',
+      ])
+  if unknown_attrs:
+    raise ValueError(
+        f'test specification {identifier} '
+        f'has unknown attributes {list(unknown_attrs)}')
+
+  for attr_type in ('mock_inputs',
+                    'mock_outputs',
+                    'expected_outputs',
+                    'expected_inputs'):
+    attr = test_spec.get(attr_type, [])
+    if not isinstance(attr, list):
+      raise TypeError(
+          f'{attr_type} of test specification {identifier} '
+          f'must be a list, got {type(attr_type)}')
+    for ix, attr_item in enumerate(attr):
+      if not isinstance(attr_item, dict):
+        raise TypeError(
+            f'{attr_type} {ix} of test specification {identifier} '
+            f'must be an object, got {type(attr_item)}')
+      if 'name' not in attr_item:
+        raise TypeError(
+            f'{attr_type} {ix} of test specification {identifier} '
+            f'missing a name')
+      if 'elements' not in attr_item:
+        raise TypeError(
+            f'{attr_type} {ix} of test specification {identifier} '
+            f'missing a elements')
+      if not isinstance(attr_item['elements'], list):
+        raise TypeError(
+            f'{attr_type} {ix} of test specification {identifier} '
+            f'must be a list, got {type(attr_item["elements"])}')
+
+
+def inject_test_tranforms(spec, test_spec, fix_failures):
+  validate_test_spec(test_spec)
+  scope = yaml_transform.LightweightScope(spec['transforms'])
+
+  mocked_inputs_by_id = {
+      scope.get_transform_id(mock_input['name']): mock_input
+      for mock_input in test_spec.get('mock_inputs', [])
+  }
+
+  mocked_outputs_by_id = _composite_key_to_nested({
+      scope.get_transform_id_and_output_name(mock_output['name']): mock_output
+      for mock_output in test_spec.get('mock_outputs', [])
+  })
+
+  recording_id_prefix = str(uuid.uuid4())
+  recording_ids = []
+
+  transforms = []
+
+  @functools.cache
+  def create_inputs(transform_id: str) -> InputsType:
+    def require_output_or_outputs(name_or_names):
+      if isinstance(name_or_names, str):
+        return require_output(name_or_names)
+      else:
+        return [require_output(name) for name in name_or_names]
+
+    if transform_id in mocked_inputs_by_id:
+      return create_mocked_input(transform_id)
+    else:
+      input_spec = scope.get_transform_spec(transform_id)['input']
+      return {
+          tag: require_output_or_outputs(input_ref)
+          for tag,
+          input_ref in yaml_transform.empty_if_explicitly_empty(
+              input_spec).items()
+      }
+
+  def require_output(name: str) -> str:
+    # The same output may be referenced under different names.
+    # Normalize before we cache.
+    transform_id, tag = scope.get_transform_id_and_output_name(name)
+    return _require_output(transform_id, tag) or name
+
+  @functools.cache
+  def _require_output(transform_id: str, tag: str) -> Optional[str]:
+    if transform_id in mocked_outputs_by_id:
+      if tag not in mocked_outputs_by_id[transform_id]:
+        name = next(iter(
+            mocked_outputs_by_id[transform_id].values()))['name'].split('.')[0]
+        raise ValueError(
+            f'Unmocked output {tag} of {name}.'
+            'If any used output is mocked all used outputs must be mocked.')
+      return create_mocked_output(transform_id, tag)
+    else:
+      _use_transform(transform_id)
+      return None  # Use original name.
+
+  @functools.cache
+  def _use_transform(transform_id: str) -> None:
+    transform_spec = dict(scope.get_transform_spec(transform_id))
+    transform_spec['input'] = create_inputs(transform_id)
+    transforms.append(transform_spec)
+
+  @functools.cache
+  def create_mocked_input(transform_id: str) -> str:
+    transform = create_create(
+        f'MockInput[{mocked_inputs_by_id[transform_id]["name"]}]',
+        mocked_inputs_by_id[transform_id]['elements'],
+        mocked_inputs_by_id[transform_id]['name'])
+    transforms.append(transform)
+    return transform['__uuid__']
+
+  @functools.cache
+  def create_mocked_output(transform_id: str, tag: str) -> str:
+    transform = create_create(
+        f'MockOutput[{mocked_outputs_by_id[transform_id][tag]["name"]}]',
+        mocked_outputs_by_id[transform_id][tag]['elements'],
+        mocked_outputs_by_id[transform_id][tag]['name'])
+    transforms.append(transform)
+    return transform['__uuid__']
+
+  def create_create(name, elements, line_source):
+    return {
+        '__uuid__': yaml_utils.SafeLineLoader.create_uuid(),
+        '__line__': yaml_utils.SafeLineLoader.get_line(line_source),
+        'name': name,
+        'type': 'Create',
+        'config': {
+            'elements': elements,
+        },
+    }
+
+  def create_assertion(name, inputs, elements, recording_id, line_source):
+    return {
+        '__uuid__': yaml_utils.SafeLineLoader.create_uuid(),
+        '__line__': yaml_utils.SafeLineLoader.get_line(line_source),
+        'name': name,
+        'input': inputs,
+        'type': 'AssertEqualAndRecord',
+        'config': {
+            'elements': elements,
+            'recording_id': recording_id,
+        },
+    }
+
+  for expected_output in test_spec.get('expected_outputs', []):
+    if fix_failures:
+      recording_id = (
+          recording_id_prefix, 'expected_outputs', expected_output['name'])
+      recording_ids.append(recording_id)
+    else:
+      recording_id = None
+    require_output(expected_output['name'])
+    transforms.append(
+        create_assertion(
+            f'CheckExpectedOutput[{expected_output["name"]}]',
+            expected_output['name'],
+            expected_output['elements'],
+            recording_id,
+            expected_output['name']))
+
+  for expected_input in test_spec.get('expected_inputs', []):
+    if fix_failures:
+      recording_id = (
+          recording_id_prefix, 'expected_inputs', expected_input['name'])
+      recording_ids.append(recording_id)
+    else:
+      recording_id = None
+    transform_id = scope.get_transform_id(expected_input['name'])
+    transforms.append(
+        create_assertion(
+            f'CheckExpectedInput[{expected_input["name"]}]',
+            create_inputs(transform_id),
+            expected_input['elements'],
+            recording_id,
+            expected_input['name']))
+
+  return {
+      '__uuid__': yaml_utils.SafeLineLoader.create_uuid(),
+      '__line__': 0,
+      'type': 'composite',
+      'transforms': transforms,
+  }, recording_ids
+
+
+class AssertEqualAndRecord(beam.PTransform):
+  _recorded_results = {}
+
+  @classmethod
+  def store_recorded_result(cls, recording_id, value):
+    assert recording_id not in cls._recorded_results
+    cls._recorded_results[recording_id] = value
+
+  @classmethod
+  def has_recorded_result(cls, recording_id):
+    return recording_id in cls._recorded_results
+
+  @classmethod
+  def get_recorded_result(cls, recording_id):
+    return cls._recorded_results[recording_id]
+
+  @classmethod
+  def remove_recorded_result(cls, recording_id):
+    del cls._recorded_results[recording_id]
+
+  def __init__(self, elements, recording_id):
+    self._elements = elements
+    self._recording_id = recording_id
+
+  def expand(self, pcoll):
+    equal_to_matcher = equal_to(yaml_provider.dicts_to_rows(self._elements))
+
+    def matcher(actual):
+      try:
+        equal_to_matcher(actual)
+      except Exception:
+        if self._recording_id:
+          AssertEqualAndRecord.store_recorded_result(
+              tuple(self._recording_id), actual)
+        else:
+          raise
+
+    return assert_that(
+        pcoll | beam.Map(lambda row: beam.Row(**row._asdict())), matcher)
+
+
+def create_test(
+    pipeline_spec, options=None, max_num_inputs=40, min_num_outputs=3):
+  if isinstance(pipeline_spec, str):
+    pipeline_spec = yaml.load(pipeline_spec, Loader=yaml_utils.SafeLineLoader)
+
+  transform_spec = _preprocess_for_testing(pipeline_spec)
+
+  if options is None:
+    options = beam.options.pipeline_options.PipelineOptions(
+        pickle_library='cloudpickle',
+        **yaml_transform.SafeLineLoader.strip_metadata(
+            pipeline_spec.get('options', {})))
+
+  def get_name(transform):
+    if 'name' in transform:
+      return str(transform['name'])
+    else:
+      if sum(1 for t in transform_spec['transforms']
+             if t['type'] == transform['type']) > 1:
+        raise ValueError('Ambiguous unnamed transform {transform["type"]}')
+      return str(transform['type'])
+
+  input_transforms = [
+      t for t in transform_spec['transforms'] if t['type'] != 'Create' and
+      not yaml_transform.empty_if_explicitly_empty(t.get('input', []))
+  ]
+
+  mock_outputs = [{
+      'name': get_name(t),
+      'elements': [
+          _try_row_as_dict(row) for row in _first_n(t, options, max_num_inputs)
+      ],
+  } for t in input_transforms]
+
+  output_transforms = [
+      t for t in transform_spec['transforms'] if t['type'] == 'LogForTesting' or
+      yaml_transform.empty_if_explicitly_empty(t.get('output', [])) or
+      t['type'].startswith('Write')
+  ]
+
+  expected_inputs = [{
+      'name': get_name(t),
+      'elements': [],
+  } for t in output_transforms]
+
+  if not expected_inputs:
+    # TODO: Optionally take this as a parameter.
+    raise ValueError('No output transforms detected.')
+
+  num_inputs = min_num_outputs
+  while True:
+    test_spec = {
+        'mock_outputs': [{
+            'name': t['name'],
+            'elements': random.sample(
+                t['elements'], min(len(t['elements']), num_inputs)),
+        } for t in mock_outputs],
+        'expected_inputs': expected_inputs,
+    }
+    fixes = run_test(pipeline_spec, test_spec, options, fix_failures=True)
+    if len(fixes) < len(output_transforms):
+      actual_output_size = 0
+    else:
+      actual_output_size = min(len(e) for e in fixes.values())
+    if actual_output_size >= min_num_outputs:
+      break
+    elif num_inputs == max_num_inputs:
+      break
+    else:
+      num_inputs = min(2 * num_inputs, max_num_inputs)
+
+  for expected_input in test_spec['expected_inputs']:
+    if ('expected_inputs', expected_input['name']) in fixes:
+      expected_input['elements'] = fixes['expected_inputs',
+                                         expected_input['name']]
+
+  return test_spec
+
+
+class _DoneException(Exception):
+  pass
+
+
+class RecordElements(beam.PTransform):
+  _recorded_results = collections.defaultdict(list)
+
+  def __init__(self, n):
+    self._n = n
+    self._id = str(uuid.uuid4())
+
+  def get_and_remove(self):
+    listing = RecordElements._recorded_results[self._id]
+    del RecordElements._recorded_results[self._id]
+    return listing
+
+  def expand(self, pcoll):
+    def record(element):
+      listing = RecordElements._recorded_results[self._id]
+      if len(listing) < self._n:
+        listing.append(element)
+      else:
+        raise _DoneException()
+
+    return pcoll | beam.Map(record)
+
+
+def _first_n(transform_spec, options, n):
+  recorder = RecordElements(n)
+  try:
+    with beam.Pipeline(options=options) as p:
+      _ = (
+          p
+          | yaml_transform.YamlTransform(
+              transform_spec,
+              providers={'AssertEqualAndRecord': AssertEqualAndRecord})
+          | recorder)
+  except _DoneException:
+    pass
+  except Exception as exn:
+    # Runners don't always raise a faithful exception type.
+    if not '_DoneException' in str(exn):
+      raise
+  return recorder.get_and_remove()
+
+
+K1 = TypeVar('K1')
+K2 = TypeVar('K2')
+V = TypeVar('V')
+
+InputsType = Dict[str, Union[str, List[str]]]
+
+
+def _composite_key_to_nested(
+    d: Mapping[Tuple[K1, K2], V]) -> Mapping[K1, Mapping[K2, V]]:
+  nested = collections.defaultdict(dict)
+  for (k1, k2), v in d.items():
+    nested[k1][k2] = v
+  return nested
+
+
+def _try_row_as_dict(row):
+  try:
+    return row._asdict()
+  except AttributeError:
+    return row
+
+
+# Linter: No need for unittest.main here.
diff --git a/sdks/python/apache_beam/yaml/yaml_testing_test.py b/sdks/python/apache_beam/yaml/yaml_testing_test.py
new file mode 100644
index 000000000000..0ff780df3614
--- /dev/null
+++ b/sdks/python/apache_beam/yaml/yaml_testing_test.py
@@ -0,0 +1,231 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import logging
+import os
+import tempfile
+import unittest
+
+# isort is fighting with yapf here.
+# isort: off
+from apache_beam.yaml import yaml_testing
+
+# Note that executing this pipeline will actually fail if the untested
+# transforms are not in fact elided.
+SIMPLE_PIPELINE = '''
+pipeline:
+  type: chain
+  transforms:
+    - type: ReadFromText
+      name: MyRead
+      config:
+        path: 'does_not_exist.txt'
+    - type: MapToFields
+      name: MyMapToFields
+      config:
+        append: True
+        language: python
+        fields:
+          square: element**2
+    - type: MapToFields
+      name: ToBeExcluded
+      config:
+        language: python
+        fields:
+          bad:
+            callable: |
+              def non_shall_pass(row):
+                raise ValueError(row)
+'''
+
+
+class YamlTestingTest(unittest.TestCase):
+  def test_expected_outputs(self):
+    yaml_testing.run_test(
+        SIMPLE_PIPELINE,
+        {
+            'mock_inputs': [{
+                'name': 'MyMapToFields',
+                'elements': [1, 2, 3],
+            }],
+            'expected_outputs': [{
+                'name': 'MyMapToFields',
+                'elements': [
+                    {
+                        'element': 1, 'square': 1
+                    },
+                    {
+                        'element': 2, 'square': 4
+                    },
+                    {
+                        'element': 3, 'square': 9
+                    },
+                ]
+            }]
+        })
+
+  def test_expected_outputs_fails_correctly(self):
+    with self.assertRaisesRegex(Exception, 'unexpected.*9'):
+      yaml_testing.run_test(
+          SIMPLE_PIPELINE,
+          {
+              'mock_inputs': [{
+                  'name': 'MyMapToFields',
+                  'elements': [1, 2, 3],
+              }],
+              'expected_outputs': [{
+                  'name': 'MyMapToFields',
+                  'elements': [
+                      {
+                          'element': 1, 'square': 1
+                      },
+                      {
+                          'element': 2, 'square': 4
+                      },
+                  ]
+              }]
+          })
+
+  def test_expected_inputs(self):
+    yaml_testing.run_test(
+        SIMPLE_PIPELINE,
+        {
+            'mock_outputs': [{
+                'name': 'MyRead',
+                'elements': [1, 2, 3],
+            }],
+            'expected_inputs': [{
+                'name': 'ToBeExcluded',
+                'elements': [
+                    {
+                        'element': 1, 'square': 1
+                    },
+                    {
+                        'element': 2, 'square': 4
+                    },
+                    {
+                        'element': 3, 'square': 9
+                    },
+                ]
+            }]
+        })
+
+  def test_expected_inputs_fails_correctly(self):
+    with self.assertRaisesRegex(Exception, 'unexpected.*9'):
+      yaml_testing.run_test(
+          SIMPLE_PIPELINE,
+          {
+              'mock_outputs': [{
+                  'name': 'MyRead',
+                  'elements': [1, 2, 3],
+              }],
+              'expected_inputs': [{
+                  'name': 'ToBeExcluded',
+                  'elements': [
+                      {
+                          'element': 1, 'square': 1
+                      },
+                      {
+                          'element': 2, 'square': 4
+                      },
+                  ]
+              }]
+          })
+
+  def test_unmocked_inputs(self):
+    with self.assertRaisesRegex(Exception, 'Non-mocked source MyRead'):
+      yaml_testing.run_test(
+          SIMPLE_PIPELINE,
+          {
+              'expected_inputs': [{
+                  'name': 'ToBeExcluded',
+                  'elements': [
+                      {
+                          'element': 1, 'square': 1
+                      },
+                      {
+                          'element': 2, 'square': 4
+                      },
+                      {
+                          'element': 3, 'square': 9
+                      },
+                  ]
+              }]
+          })
+
+  def test_fixes(self):
+    fixes = yaml_testing.run_test(
+        SIMPLE_PIPELINE,
+        {
+            'mock_outputs': [{
+                'name': 'MyRead',
+                'elements': [1, 2, 3],
+            }],
+            'expected_inputs': [{
+                'name': 'ToBeExcluded',
+                'elements': [
+                    {
+                        'element': 1, 'square': 1
+                    },
+                    {
+                        'element': 2, 'square': 4
+                    },
+                ]
+            }]
+        },
+        fix_failures=True)
+    self.assertEqual(
+        fixes,
+        {('expected_inputs', 'ToBeExcluded'): [
+             dict(element=1, square=1),
+             dict(element=2, square=4),
+             dict(element=3, square=9),
+         ]})
+
+  def test_create(self):
+    with tempfile.TemporaryDirectory() as tmpdir:
+      input_path = os.path.join(tmpdir, 'input.csv')
+      input_path = os.path.join('.', 'input.csv')
+      with open(input_path, 'w') as fout:
+        fout.write('a,b,c\n')
+        for ix in range(1000):
+          fout.write(f'{ix % 5},{ix},Ccc\n')
+      pipeline = f'''
+      pipeline:
+        type: chain
+        transforms:
+          - type: ReadFromCsv
+            config:
+              path: {input_path}
+          - type: Filter
+            config:
+              language: python
+              keep: a == 1
+          - type: WriteToSink
+      '''
+      test_spec = yaml_testing.create_test(
+          pipeline, max_num_inputs=100, min_num_outputs=5)
+
+    self.assertEqual(len(test_spec['mock_outputs']), 1)
+    self.assertEqual(len(test_spec['expected_inputs']), 1)
+    self.assertGreaterEqual(len(test_spec['expected_inputs'][0]['elements']), 5)
+    yaml_testing.run_test(pipeline, test_spec)
+
+
+if __name__ == '__main__':
+  logging.getLogger().setLevel(logging.INFO)
+  unittest.main()
diff --git a/sdks/python/apache_beam/yaml/yaml_transform.py b/sdks/python/apache_beam/yaml/yaml_transform.py
index 744cbe6a8925..09eee94d7cd6 100644
--- a/sdks/python/apache_beam/yaml/yaml_transform.py
+++ b/sdks/python/apache_beam/yaml/yaml_transform.py
@@ -161,6 +161,9 @@ def get_transform_id(self, transform_name):
       else:
         return only_element(candidates)
 
+  def get_transform_spec(self, transform_name_or_id):
+    return self._transforms_by_uuid[self.get_transform_id(transform_name_or_id)]
+
 
 class Scope(LightweightScope):
   """To look up PCollections (typically outputs of prior transforms) by name."""
@@ -573,9 +576,10 @@ def is_not_output_of_last_transform(new_transforms, value):
     raise TypeError(
         f"Chain at {identify_object(spec)} missing transforms property.")
   has_explicit_outputs = 'output' in spec
-  composite_spec = normalize_inputs_outputs(tag_explicit_inputs(spec))
+  composite_spec = dict(normalize_inputs_outputs(tag_explicit_inputs(spec)))
   new_transforms = []
   for ix, transform in enumerate(composite_spec['transforms']):
+    transform = dict(transform)
     if any(io in transform for io in ('input', 'output')):
       if (ix == 0 and 'input' in transform and 'output' not in transform and
           is_explicitly_empty(transform['input'])):
@@ -939,17 +943,18 @@ def ensure_config(spec):
   return spec
 
 
+def apply_phase(phase, spec):
+  spec = phase(spec)
+  if spec['type'] in {'composite', 'chain'} and 'transforms' in spec:
+    spec = dict(
+        spec, transforms=[apply_phase(phase, t) for t in spec['transforms']])
+  return spec
+
+
 def preprocess(spec, verbose=False, known_transforms=None):
   if verbose:
     pprint.pprint(spec)
 
-  def apply(phase, spec):
-    spec = phase(spec)
-    if spec['type'] in {'composite', 'chain'} and 'transforms' in spec:
-      spec = dict(
-          spec, transforms=[apply(phase, t) for t in spec['transforms']])
-    return spec
-
   if known_transforms:
     known_transforms = set(known_transforms).union(['chain', 'composite'])
 
@@ -1012,7 +1017,7 @@ def validate_transform_references(spec):
       # lift_config,
       ensure_config,
   ]:
-    spec = apply(phase, spec)
+    spec = apply_phase(phase, spec)
     if verbose:
       print('=' * 20, phase, '=' * 20)
       pprint.pprint(spec)
diff --git a/sdks/python/apache_beam/yaml/yaml_utils.py b/sdks/python/apache_beam/yaml/yaml_utils.py
index 926054ffe037..ce71c2d7a31e 100644
--- a/sdks/python/apache_beam/yaml/yaml_utils.py
+++ b/sdks/python/apache_beam/yaml/yaml_utils.py
@@ -15,11 +15,16 @@
 # limitations under the License.
 #
 
+import json
 import os
+import re
 import uuid
 from collections.abc import Iterable
 from collections.abc import Mapping
+from typing import Any
+from typing import Tuple
 
+import yaml
 from yaml import SafeLoader
 
 
@@ -76,5 +81,164 @@ def get_line(obj):
       return getattr(obj, '_line_', 'unknown')
 
 
+def patch_yaml(original_str: str, updated):
+  """Updates a yaml string to match the updated with minimal changes.
+
+  This only changes the portions of original_str that differ between
+  original_str and updated in an attempt to preserve comments and formatting.
+  """
+  if not original_str and updated:
+    return yaml.dump(updated, sort_keys=False)
+
+  if original_str[-1] != '\n':
+    # Add a trialing newline to avoid having to constantly check this edge case.
+    # (It's also a good idea generally...)
+    original_str += '\n'
+
+  # The yaml parser returns positions in terms of line and column numbers.
+  # Here we construct the mapping between the two.
+  line_starts = [0]
+  ix = original_str.find('\n')
+  while ix != -1:
+    line_starts.append(ix + 1)
+    ix = original_str.find('\n', ix + 1)
+
+  def pos(line_or_mark, column=0):
+    if isinstance(line_or_mark, yaml.Mark):
+      line = line_or_mark.line
+      column = line_or_mark.column
+    else:
+      line = line_or_mark
+    return line_starts[line] + column
+
+  # Here we define a custom loader with hooks that record where each element is
+  # found so we can swap it out appropriately.
+  spans = {}
+
+  class SafeMarkLoader(SafeLoader):
+    pass
+
+  # We create special subclass types to ensure each returned node is
+  # a distinct object.
+  marked_types = {}
+
+  def record_yaml_scalar(constructor):
+    def wrapper(self, node):
+      raw_data = constructor(self, node)
+      typ = type(raw_data)
+      if typ not in marked_types:
+        marked_types[typ] = type(f'Marked_{typ}', (type(raw_data), ), {})
+      marked_data = marked_types[typ](raw_data)
+      spans[id(marked_data)] = node.start_mark, node.end_mark
+      return marked_data
+
+    return wrapper
+
+  SafeMarkLoader.add_constructor(
+      'tag:yaml.org,2002:seq',
+      record_yaml_scalar(SafeMarkLoader.construct_sequence))
+  SafeMarkLoader.add_constructor(
+      'tag:yaml.org,2002:map',
+      record_yaml_scalar(SafeMarkLoader.construct_mapping))
+  for typ in ('bool', 'int', 'float', 'binary', 'timestamp', 'str'):
+    SafeMarkLoader.add_constructor(
+        f'tag:yaml.org,2002:{typ}',
+        record_yaml_scalar(getattr(SafeMarkLoader, f'construct_yaml_{typ}')))
+
+  #  Now load the original yaml using our special parser.
+  original = yaml.load(original_str, Loader=SafeMarkLoader)
+
+  # This (recursively) finds the portion of the original string that must
+  # be replaced with new content.
+  def diff(a: Any, b: Any) -> Iterable[Tuple[int, int, str]]:
+    if a == b:
+      return
+    elif (isinstance(a, dict) and isinstance(b, dict) and
+          set(a.keys()) == set(b.keys()) and
+          all(id(v) in spans for v in a.values())):
+      for k, v in a.items():
+        yield from diff(v, b[k])
+    elif (isinstance(a, list) and isinstance(b, list) and a and b and
+          all(id(v) in spans for v in a)):
+      # Diff the matching entries.
+      for va, vb in zip(a, b):
+        yield from diff(va, vb)
+      if len(b) < len(a):
+        # Remove extra entries
+        yield (
+            # End of last preserved element.
+            pos(spans[id(a[len(b) - 1])][1]),
+            # End of last original element.
+            pos(spans[id(a[-1])][1]),
+            '')
+      elif len(b) > len(a):
+        # Add extra entries
+        list_start, list_end = spans[id(a)]
+        start_char = original_str[pos(list_start)]
+        if start_char == '[':
+          for v in b[len(a):]:
+            yield pos(list_end) - 1, pos(list_end) - 1, ', ' + json.dumps(v)
+        else:
+          assert start_char == '-'
+          indent = original_str[pos(list_start.line):pos(list_start)] + '- '
+          content = original_str[pos(list_start):pos(list_end)].rstrip()
+          actual_end_pos = pos(list_start) + len(content)
+          for v in b[len(a):]:
+            if isinstance(v, (list, dict)):
+              v_str = (
+                  yaml.dump(v, sort_keys=False)
+                  # Indent.
+                  .replace('\n', '\n' + ' ' * len(indent))
+                  # Remove blank line indents.
+                  .replace(' ' * len(indent) + '\n', '\n').rstrip())
+            else:
+              v_str = json.dumps(v)
+            yield actual_end_pos, actual_end_pos, '\n' + indent + v_str
+
+    else:
+      start, end = spans[id(a)]
+      indent = original_str[pos(start.line):pos(start)]
+      # We strip trailing whitespace as the "end" of an element is often on
+      # a subsequent line where the subsequent element actually starts.
+      content = original_str[pos(start):pos(end)].rstrip()
+      actual_end_pos = pos(start) + len(content)
+      trailing = original_str[actual_end_pos:original_str.
+                              find('\n', actual_end_pos)]
+      if isinstance(b, (list, dict)):
+        if indent.strip() in ('', '-') and not trailing.strip():
+          # This element wholly occupies its set of lines, so it is safe to use
+          # a multi-line yaml representation (appropriately indented).
+          yield (
+              pos(start),
+              actual_end_pos,
+              yaml.dump(b, sort_keys=False)
+              # Indent.
+              .replace('\n', '\n' + ' ' * len(indent))
+              # Remove blank line indents.
+              .replace(' ' * len(indent) + '\n', '\n').rstrip())
+        else:
+          # Force flow style.
+          yield (
+              pos(start),
+              actual_end_pos,
+              yaml.dump(b, default_flow_style=True, line_break=False).strip())
+      elif isinstance(b, str) and re.match('^[A-Za-z0-9_]+$', b):
+        # A simple string literal.
+        yield pos(start), actual_end_pos, b
+      else:
+        # A scalar.
+        yield pos(start), actual_end_pos, json.dumps(b)
+
+  # Now stick it all together.
+  last_end = 0
+  content = []
+  for start, end, new_content in sorted(diff(original, updated)):
+    content.append(original_str[last_end:start])
+    content.append(new_content)
+    last_end = end
+  content.append(original_str[last_end:])
+  return ''.join(content)
+
+
 def locate_data_file(relpath):
   return os.path.join(os.path.dirname(__file__), relpath)
diff --git a/sdks/python/apache_beam/yaml/yaml_utils_test.py b/sdks/python/apache_beam/yaml/yaml_utils_test.py
index 4fd2c793e57e..353d95fd8173 100644
--- a/sdks/python/apache_beam/yaml/yaml_utils_test.py
+++ b/sdks/python/apache_beam/yaml/yaml_utils_test.py
@@ -21,6 +21,7 @@
 import yaml
 
 from apache_beam.yaml.yaml_utils import SafeLineLoader
+from apache_beam.yaml.yaml_utils import patch_yaml
 
 
 class SafeLineLoaderTest(unittest.TestCase):
@@ -74,6 +75,145 @@ def test_strip_metadata_nothing_to_strip(self):
     self.assertFalse(hasattr(stripped, '__uuid__'))
 
 
+_BASIC_YAML = '''
+
+a: [1, 2, 3]
+# comment
+
+b:
+  - 1
+  - 2
+  - 3
+
+outer:
+  - k1: v1
+    k2: v2
+    k3: [1, 2, {x: y}]
+    k0: out_of_order
+  - item
+
+'''
+
+
+class PatchYamlTest(unittest.TestCase):
+  def test_unchanged(self):
+    obj = yaml.load(_BASIC_YAML, Loader=yaml.SafeLoader)
+    self.assertEqual(patch_yaml(_BASIC_YAML, obj), _BASIC_YAML)
+
+  def test_inline_list(self):
+    obj = yaml.load(_BASIC_YAML, Loader=yaml.SafeLoader)
+    obj['a'] = [100, 200]
+    self.assertEqual(
+        patch_yaml(_BASIC_YAML, obj),
+        _BASIC_YAML.replace('[1, 2, 3]', '[100, 200]'))
+
+  def test_bullet_list(self):
+    obj = yaml.load(_BASIC_YAML, Loader=yaml.SafeLoader)
+    obj['b'] = [100, 200]
+    self.assertEqual(
+        patch_yaml(_BASIC_YAML, obj),
+        _BASIC_YAML.replace('''
+  - 1
+  - 2
+  - 3
+''', '''
+  - 100
+  - 200
+'''))
+
+  def test_inline_map(self):
+    obj = yaml.load(_BASIC_YAML, Loader=yaml.SafeLoader)
+    obj['outer'][0]['k3'][-1] = 500
+    self.assertEqual(
+        patch_yaml(_BASIC_YAML, obj), _BASIC_YAML.replace('{x: y}', '500'))
+
+  def test_multiline_map(self):
+    obj = yaml.load(_BASIC_YAML, Loader=yaml.SafeLoader)
+    obj['outer'][0]['k4'] = 500
+    self.assertEqual(
+        patch_yaml(_BASIC_YAML, obj),
+        _BASIC_YAML.replace(
+            '''
+  - k1: v1
+    k2: v2
+    k3: [1, 2, {x: y}]
+    k0: out_of_order
+'''.strip(),
+            '''
+  - k1: v1
+    k2: v2
+    k3:
+    - 1
+    - 2
+    - x: y
+    k0: out_of_order
+    k4: 500
+'''.strip()))
+
+  def test_inline_list_shorten(self):
+    obj = yaml.load(_BASIC_YAML, Loader=yaml.SafeLoader)
+    obj['a'] = [100, 2]
+    self.assertEqual(
+        patch_yaml(_BASIC_YAML, obj),
+        _BASIC_YAML.replace('[1, 2, 3]', '[100, 2]'))
+
+  def test_bullet_list_shorten(self):
+    obj = yaml.load(_BASIC_YAML, Loader=yaml.SafeLoader)
+    obj['b'] = [100, 2]
+    self.assertEqual(
+        patch_yaml(_BASIC_YAML, obj),
+        _BASIC_YAML.replace(
+            '''
+  - 1
+  - 2
+  - 3
+        '''.strip(),
+            '''
+  - 100
+  - 2
+        '''.strip()))
+
+  def test_inline_list_extend(self):
+    obj = yaml.load(_BASIC_YAML, Loader=yaml.SafeLoader)
+    obj['a'] = [100, 2, 3, 4]
+    self.assertEqual(
+        patch_yaml(_BASIC_YAML, obj),
+        _BASIC_YAML.replace('[1, 2, 3]', '[100, 2, 3, 4]'))
+
+  def test_bullet_list_extend(self):
+    obj = yaml.load(_BASIC_YAML, Loader=yaml.SafeLoader)
+    obj['b'] = [100, 2, 3, 4]
+    self.assertEqual(
+        patch_yaml(_BASIC_YAML, obj),
+        _BASIC_YAML.replace(
+            '''
+  - 1
+  - 2
+  - 3
+        '''.strip(),
+            '''
+  - 100
+  - 2
+  - 3
+  - 4
+        '''.strip()))
+
+  def test_bullet_list_map_extend(self):
+    obj = yaml.load(_BASIC_YAML, Loader=yaml.SafeLoader)
+    obj['outer'].append({'a': 'A', 'b': 'B'})
+    self.assertEqual(
+        patch_yaml(_BASIC_YAML, obj),
+        _BASIC_YAML.replace(
+            '''
+  - item
+        '''.strip(),
+            '''
+  - item
+  - a: A
+    b: B
+        '''.strip()))
+
+
 if __name__ == '__main__':
   logging.getLogger().setLevel(logging.INFO)
   unittest.main()
diff --git a/sdks/python/scripts/run_pylint.sh b/sdks/python/scripts/run_pylint.sh
index 9ea8b31d62fc..644b30fba66f 100755
--- a/sdks/python/scripts/run_pylint.sh
+++ b/sdks/python/scripts/run_pylint.sh
@@ -102,6 +102,8 @@ ISORT_EXCLUDED=(
   "taxi.py"
   "tfdv_analyze_and_validate.py"
   "yaml/main.py"
+  "main_test.py"
+  "yaml_testing_test.py"
 )
 SKIP_PARAM=""
 for file in "${ISORT_EXCLUDED[@]}"; do
diff --git a/website/www/site/content/en/documentation/sdks/yaml-testing.md b/website/www/site/content/en/documentation/sdks/yaml-testing.md
new file mode 100644
index 000000000000..4066e18f5ec5
--- /dev/null
+++ b/website/www/site/content/en/documentation/sdks/yaml-testing.md
@@ -0,0 +1,303 @@
+---
+type: languages
+title: "Apache Beam YAML Testing"
+---
+<!--
+    Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+-->
+
+# Beam YAML Tests
+
+A robust testing story is an important part of any production setup.
+Though the various built-in (and externally provided) transform in a Beam YAML
+pipeline can be expected to be well tested, it can be important to have tests
+that ensure the pipeline as a whole behaves as expected.  This is particularly
+true for transforms that contain non-trivial UDF logic.
+
+# Whole pipeline tests
+
+For example, consider the example word count pipeline.
+
+```
+pipeline:
+  transforms:
+  - type: ReadFromText
+    name: Read from GCS
+    config:
+      path: gs://dataflow-samples/shakespeare/kinglear.txt
+  - type: MapToFields
+    name: Split words
+    config:
+      language: python
+      fields:
+        word:
+          callable: |
+            import re
+            def all_words(row):
+              return re.findall(r'[a-z]+', row.line.lower())
+        value: 1
+    input: Read from GCS
+  - type: Explode
+    name: Explode word arrays
+    config:
+      fields: [word]
+    input: Split words
+  - type: Combine
+    name: Count words
+    config:
+      group_by: [word]
+      combine:
+        value: sum
+    input: Explode word arrays
+  - type: MapToFields
+    name: Format output
+    config:
+      language: python
+      fields:
+        output: "word + ': ' + str(value)"
+    input: Count words
+  - type: WriteToText
+    name: Write to GCS
+    config:
+      path: gs://bucket/counts.txt
+    input: Format output
+
+tests: []
+```
+
+To write tests for this pipeline, one creates a `tests` section that enumerates
+a number of tests, each of which provide example input and assert the expected
+output is produced.  An example test might be as follows
+
+```
+tests:
+- name: MyRegressionTest
+  mock_outputs:
+    - name: Read from GCS
+      elements:
+        - line: "Nothing can come of nothing"
+  expected_inputs:
+    - name: Write to GCS
+      elements:
+        - output: 'nothing: 2'
+        - output: 'can: 1'
+        - output: 'come: 1'
+        - output: 'of: 1'
+```
+
+The `mock_outputs` section designates that the transform named `Read from GCS`
+should produce the single row `{line: "Nothing can come of nothing"}` for the
+purposes of this test, and the `expected_inputs` section indicates that the
+transform `Write to GCS` should expect to receive exactly the given elements.
+Neither the actual Read transform nor Write transform from the original
+pipelines are executed when running the test, but all intermediate transforms
+are.
+
+This test can then be executed by running
+
+```
+python -m apache_beam.yaml.main \
+    --yaml_pipeline_file=wordcount.yaml \
+    --tests
+```
+
+Alternatively, the a `tests:` block may be placed in a separate file and be
+validated by running
+
+```
+python -m apache_beam.yaml.main \
+    --yaml_pipeline_file=wordcount.yaml \
+    --tests \
+    --test_suite=test_file.yaml
+```
+
+Neither the actual Read transform nor Write transform from the original
+pipelines are executed when running the test, but all intermediate transforms
+are.  For hermeticity, we require that all inputs (with the exception of
+`Create` that are needed to compute the expected outputs are explicitly mocked;
+to explicitly allow a sources to be executed as part of a test their names or
+types can be enumerated in an `allowed_sources` attribute of the test
+specification.
+
+
+## Pipeline fragment tests
+
+One can also tests a portion of a pipeline using the `mock_inputs` and
+`expected_outputs` section of a test, for example
+
+```
+tests:
+- name: TestSplittingWithPunctuation
+  mock_inputs:
+    - name: Split words
+      elements:
+        - line: "lots-of-words"
+        - line: "...and more"
+  expected_outputs:
+    - name: Explode
+      elements:
+        - word: lots
+          value: 1
+        - word: of
+          value: 1
+        - word: words
+          value: 1
+        - word: and
+          value: 1
+        - word: more
+          value: 1
+
+- name: TestCombineAndFormat
+  mock_inputs:
+    - name: Count words
+      elements:
+        - word: more
+          value: 1
+        - word: and
+          value: 1
+        - word: more
+          value: 1
+  expected_outputs:
+    - name: Format output
+      elements:
+        - output: "more: 2"
+        - output: "and: 1"
+```
+
+As before, each test only executes the portion of the pipeline between the
+mock inputs and expected outputs.  Note that the named transform in a
+`mock_inputs` specification *is* executed, while the named transform of a
+`mock_oupputs` specification is not.
+Similarly, the named transform of a `expected_inputs` specification is *not*
+executed, while the named transform of an `expected_outputs` necessarily is.
+
+
+## Automatically generating tests.
+
+In an effort to make tests as easy to write and maintain as possible,
+Beam YAML provides utilities to compute the expected outputs for your tests.
+
+
+Running
+
+```
+python -m apache_beam.yaml.main \
+    --yaml_pipeline_file=wordcount.yaml \
+    --tests \
+    [--test_suite=...] \
+    --create_test
+```
+
+will create an entirely new test by sampling all the sources and
+constructing a test accordingly.
+
+One can also keep tests up to date by running
+
+```
+python -m apache_beam.yaml.main \
+    --yaml_pipeline_file=wordcount.yaml \
+    --tests \
+    [--test_suite=...] \
+    --fix_tests
+```
+
+which will update any existing `expected_input` and `expected_output` blocks
+of your pipeline to contain the actual values computed during the test.
+This can be useful in authoring tests as well--one can simply specify a
+nonsensical or empty elements block in the expectation and the `--fix_tests`
+flag will populate it for you.
+(Of course, it is on any user of these flags to verify that the produced values
+are meaningful and as expected.)
+
+
+## Branching pipelines
+
+For complex, branching pipelines, any number of `mock_inputs` and `mock_outupts`
+may be enumerated to provide the input data, and any number of `expected_inputs`
+and `expected_outputs` validations may be specified as well.
+In both the `mock_outupts` and `expected_outputs` block, multiple outputs can
+be disambiguated with the `TransformName.output_name` notation just as when
+authoring a yaml pipeline.
+
+```
+pipeline:
+  transforms:
+    - type: Create
+      name: Abc
+      config:
+        elements: [a, b, ccc]
+    - type: Create
+      name: Xyz
+      config:
+        elements: [x, y, zzz]
+    - type: MapToFields
+      name: Upper
+      input: [Abc, Xyz]
+      config:
+        language: python
+        fields:
+          element: element.upper()
+    - type: Partition
+      input: Upper
+      config:
+        language: python
+        by: '"big" if len(element) > 1 else "small"'
+        outputs: ["big", "small"]
+    - type: MapToFields
+      name: MaybeHasErrors
+      input: Abc
+      config:
+        language: python
+        fields:
+          inverse_size: 1 / len(element)
+        error_handling:
+          output: errors
+    - type: StripErrorMetadata
+      input: MaybeHasErrors.errors
+
+tests:
+  - name: MockMultipleInputs
+    mock_outputs:
+      - name: Abc
+        elements: [element: a]
+      - name: Xyz
+        elements: [element: z]
+    expected_outputs:
+      - name: Upper
+        elements: [element: A, element: Z]
+
+  - name: TestMultipelOuptuts
+    mock_inputs:
+      - name: Upper
+        elements: [element: m, element: nnn]
+    expected_outputs:
+      - name: Partition.big
+        elements: [element: NNN]
+      - name: Partition.small
+        elements: [element: M]
+
+  - name: TestErrorHandling
+    mock_outputs:
+      - name: Abc
+        elements: [element: 'Aaaa', element: '']
+    expected_outputs:
+      - name: MaybeHasErrors
+        elements: [inverse_size: 0.25]
+      - name: StripErrorMetadata
+        elements: [element: '']
+```
diff --git a/website/www/site/content/en/documentation/sdks/yaml-udf.md b/website/www/site/content/en/documentation/sdks/yaml-udf.md
index a3e4790c7b34..69bdc77704d1 100644
--- a/website/www/site/content/en/documentation/sdks/yaml-udf.md
+++ b/website/www/site/content/en/documentation/sdks/yaml-udf.md
@@ -336,6 +336,9 @@ It is also possible to store the function logic in a file and point to the funct
       name: my_filter
 ```
 
+This allows the logic of the UDF itself to be more easily developed and tested
+using standard software engineering practices.
+
 Currently, in addition to Python, Java, SQL, and JavaScript (experimental)
 expressions are supported as well
 
diff --git a/website/www/site/layouts/partials/section-menu/en/sdks.html b/website/www/site/layouts/partials/section-menu/en/sdks.html
index 38c014617706..4766ea502b56 100644
--- a/website/www/site/layouts/partials/section-menu/en/sdks.html
+++ b/website/www/site/layouts/partials/section-menu/en/sdks.html
@@ -93,6 +93,7 @@
     <li><a href="/documentation/sdks/yaml-udf/">Yaml User Defined Functions</a></li>
     <li><a href="/documentation/sdks/yaml-combine/">Yaml Aggregation</a></li>
     <li><a href="/documentation/sdks/yaml-errors/">Error handling</a></li>
+    <li><a href="/documentation/sdks/yaml-testing/">Yaml Testing</a></li>
     <li><a href="/documentation/sdks/yaml-inline-python/">Inlining Python</a></li>
     <li><a href="/documentation/sdks/yaml-providers/">Yaml Providers</a></li>
     <li><a href="/documentation/sdks/yaml-join/">Yaml Join</a></li>


Reviews:
user: login: damccorm
body: Thanks! Had some minor comments, but overall it looks good to me
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T15:10:16Z
user: login: robertwb
body: Thanks for looking at this! Given your comments are throughout the files, should I assume you're reviewing the entire change? (I tried to break it up into pull reviews both to get it out there sooner and spread the load, but maybe that's not to helpful after all.)
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T01:19:16Z
user: login: damccorm
body: Thanks! Could you fix the linting check? Otherwise, it LGTM
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T14:29:24Z

Review comments:
diff_hunk: @@ -130,12 +155,112 @@ def run(argv=None):
       print('Running pipeline...')
 
 
-def build_pipeline_components_from_argv(argv):
+def run_tests(argv=None, exit=True):
+  known_args, pipeline_args, _, pipeline_yaml = _build_pipeline_yaml_from_argv(
+      argv)
+  pipeline_spec = yaml.load(pipeline_yaml, Loader=yaml_transform.SafeLineLoader)
+  options = _build_pipeline_options(pipeline_spec, pipeline_args)
+
+  if known_args.create_test and not known_args.fix_tests:
+    result = unittest.TestResult()
+    tests = []
+  else:
+    if known_args.test_suite:
+      with open(known_args.test_suite) as fin:
+        test_suite = yaml.load(fin, Loader=yaml_transform.SafeLineLoader) or {}
+      if 'tests' not in test_suite or not isinstance(test_suite['tests'], list):
+        raise TypeError('tests attribute must be a list of test specifications')
path: sdks/python/apache_beam/yaml/main.py
user: login: damccorm
body: ```suggestion
        raise TypeError('Tests attribute must be a list of test specifications. If you haven't added a set of tests yet, you can get started by running your pipeline with the --create_test flag enabled')
```

Thoughts on something like this? (suggestion can't be accepted as is because of linting)
created_at: 2025-04-08T14:42:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -130,12 +155,112 @@ def run(argv=None):
       print('Running pipeline...')
 
 
-def build_pipeline_components_from_argv(argv):
+def run_tests(argv=None, exit=True):
+  known_args, pipeline_args, _, pipeline_yaml = _build_pipeline_yaml_from_argv(
+      argv)
+  pipeline_spec = yaml.load(pipeline_yaml, Loader=yaml_transform.SafeLineLoader)
+  options = _build_pipeline_options(pipeline_spec, pipeline_args)
+
+  if known_args.create_test and not known_args.fix_tests:
path: sdks/python/apache_beam/yaml/main.py
user: login: damccorm
body: Can we just make `fix_tests` incompatible with the other test args (and throw if violated) rather than special casing it here? It seems like it should always be its own thing.
created_at: 2025-04-08T14:46:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -130,12 +155,112 @@ def run(argv=None):
       print('Running pipeline...')
 
 
-def build_pipeline_components_from_argv(argv):
+def run_tests(argv=None, exit=True):
+  known_args, pipeline_args, _, pipeline_yaml = _build_pipeline_yaml_from_argv(
+      argv)
+  pipeline_spec = yaml.load(pipeline_yaml, Loader=yaml_transform.SafeLineLoader)
+  options = _build_pipeline_options(pipeline_spec, pipeline_args)
+
+  if known_args.create_test and not known_args.fix_tests:
+    result = unittest.TestResult()
+    tests = []
+  else:
+    if known_args.test_suite:
+      with open(known_args.test_suite) as fin:
+        test_suite = yaml.load(fin, Loader=yaml_transform.SafeLineLoader) or {}
+      if 'tests' not in test_suite or not isinstance(test_suite['tests'], list):
+        raise TypeError('tests attribute must be a list of test specifications')
path: sdks/python/apache_beam/yaml/main.py
user: login: damccorm
body: Same suggestion applies below as well
created_at: 2025-04-08T14:47:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -90,6 +95,26 @@ def _parse_arguments(argv):
       type=json.loads,
       help='A json dict of variables used when invoking the jinja preprocessor '
       'on the provided yaml pipeline.')
+  parser.add_argument(
+      '--test',
+      action=argparse.BooleanOptionalAction,
+      help='Run the tests associated with the given pipeline, rather than the '
+      'pipeline itself.')
+  parser.add_argument(
+      '--fix_tests',
+      action=argparse.BooleanOptionalAction,
+      help='Update failing test expectations to match the actual ouput.')
path: sdks/python/apache_beam/yaml/main.py
user: login: damccorm
body: Could we mention that this requires the `test-suite` flag to be specified as of now
created_at: 2025-04-08T14:51:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -90,6 +95,26 @@ def _parse_arguments(argv):
       type=json.loads,
       help='A json dict of variables used when invoking the jinja preprocessor '
       'on the provided yaml pipeline.')
+  parser.add_argument(
+      '--test',
+      action=argparse.BooleanOptionalAction,
+      help='Run the tests associated with the given pipeline, rather than the '
+      'pipeline itself.')
+  parser.add_argument(
+      '--fix_tests',
+      action=argparse.BooleanOptionalAction,
+      help='Update failing test expectations to match the actual ouput.')
path: sdks/python/apache_beam/yaml/main.py
user: login: damccorm
body: Could we mention that this requires the `test-suite` flag to be specified as of now
created_at: 2025-04-08T14:51:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -130,12 +155,112 @@ def run(argv=None):
       print('Running pipeline...')
 
 
-def build_pipeline_components_from_argv(argv):
+def run_tests(argv=None, exit=True):
+  known_args, pipeline_args, _, pipeline_yaml = _build_pipeline_yaml_from_argv(
+      argv)
+  pipeline_spec = yaml.load(pipeline_yaml, Loader=yaml_transform.SafeLineLoader)
+  options = _build_pipeline_options(pipeline_spec, pipeline_args)
+
+  if known_args.create_test and not known_args.fix_tests:
+    result = unittest.TestResult()
+    tests = []
+  else:
+    if known_args.test_suite:
+      with open(known_args.test_suite) as fin:
+        test_suite = yaml.load(fin, Loader=yaml_transform.SafeLineLoader) or {}
+      if 'tests' not in test_suite or not isinstance(test_suite['tests'], list):
+        raise TypeError('tests attribute must be a list of test specifications')
+      test_specs = test_suite['tests']
+    else:
+      test_specs = pipeline_spec.get('tests', [])
+      if not isinstance(test_specs, list):
+        raise TypeError('tests attribute must be a list of test specifications')
+    if not test_specs:
+      raise RuntimeError('No tests found.')
+
+    with _fix_xlang_instant_coding():
+      tests = [
+          _YamlTestCase(
+              pipeline_spec, test_spec, options, known_args.fix_tests)
+          for test_spec in test_specs
+      ]
+      suite = unittest.TestSuite(tests)
+      result = unittest.TextTestRunner().run(suite)
+
+  if known_args.fix_tests or known_args.create_test:
+    update_tests(known_args, pipeline_spec, options, tests)
+
+  if exit:
+    # emulates unittest.main()
+    sys.exit(0 if result.wasSuccessful() else 1)
+  else:
+    if not result.wasSuccessful():
+      raise RuntimeError(result)
+
+
+def update_tests(known_args, pipeline_spec, options, tests):
+  if known_args.test_suite:
+    path = known_args.test_suite
+    if not os.path.exists(path) and known_args.create_test:
+      with open(path, 'w') as fout:
+        fout.write('tests: []')
+  elif known_args.yaml_pipeline_file:
+    path = known_args.yaml_pipeline_file
+  else:
+    raise RuntimeError(
+        'Test fixing only supported for file-backed tests. '
+        'Please use the --test_suite flag.')
+  with open(path) as fin:
+    original_yaml = fin.read()
+  if path == known_args.yaml_pipeline_file and pipeline_yaml.strip(
+  ) != original_yaml.strip():
+    raise RuntimeError(
+        'In-file test fixing not yet supported for templated pipelines. '
+        'Please use the --test_suite flag.')
+  updated_spec = yaml.load(original_yaml, Loader=yaml.SafeLoader) or {}
+
+  if known_args.fix_tests:
+    for ix, test in enumerate(tests):
+      if test.fixes:
path: sdks/python/apache_beam/yaml/main.py
user: login: damccorm
body: Won't this always be true at this point?
created_at: 2025-04-08T14:52:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -130,12 +155,112 @@ def run(argv=None):
       print('Running pipeline...')
 
 
-def build_pipeline_components_from_argv(argv):
+def run_tests(argv=None, exit=True):
+  known_args, pipeline_args, _, pipeline_yaml = _build_pipeline_yaml_from_argv(
+      argv)
+  pipeline_spec = yaml.load(pipeline_yaml, Loader=yaml_transform.SafeLineLoader)
+  options = _build_pipeline_options(pipeline_spec, pipeline_args)
+
+  if known_args.create_test and not known_args.fix_tests:
+    result = unittest.TestResult()
+    tests = []
+  else:
+    if known_args.test_suite:
+      with open(known_args.test_suite) as fin:
+        test_suite = yaml.load(fin, Loader=yaml_transform.SafeLineLoader) or {}
+      if 'tests' not in test_suite or not isinstance(test_suite['tests'], list):
+        raise TypeError('tests attribute must be a list of test specifications')
+      test_specs = test_suite['tests']
+    else:
+      test_specs = pipeline_spec.get('tests', [])
+      if not isinstance(test_specs, list):
+        raise TypeError('tests attribute must be a list of test specifications')
+    if not test_specs:
+      raise RuntimeError('No tests found.')
+
+    with _fix_xlang_instant_coding():
+      tests = [
+          _YamlTestCase(
+              pipeline_spec, test_spec, options, known_args.fix_tests)
+          for test_spec in test_specs
+      ]
+      suite = unittest.TestSuite(tests)
+      result = unittest.TextTestRunner().run(suite)
+
+  if known_args.fix_tests or known_args.create_test:
+    update_tests(known_args, pipeline_spec, options, tests)
+
+  if exit:
+    # emulates unittest.main()
+    sys.exit(0 if result.wasSuccessful() else 1)
+  else:
+    if not result.wasSuccessful():
+      raise RuntimeError(result)
+
+
+def update_tests(known_args, pipeline_spec, options, tests):
+  if known_args.test_suite:
+    path = known_args.test_suite
+    if not os.path.exists(path) and known_args.create_test:
+      with open(path, 'w') as fout:
+        fout.write('tests: []')
+  elif known_args.yaml_pipeline_file:
+    path = known_args.yaml_pipeline_file
+  else:
+    raise RuntimeError(
+        'Test fixing only supported for file-backed tests. '
+        'Please use the --test_suite flag.')
+  with open(path) as fin:
+    original_yaml = fin.read()
+  if path == known_args.yaml_pipeline_file and pipeline_yaml.strip(
+  ) != original_yaml.strip():
+    raise RuntimeError(
+        'In-file test fixing not yet supported for templated pipelines. '
+        'Please use the --test_suite flag.')
+  updated_spec = yaml.load(original_yaml, Loader=yaml.SafeLoader) or {}
+
+  if known_args.fix_tests:
+    for ix, test in enumerate(tests):
+      if test.fixes:
path: sdks/python/apache_beam/yaml/main.py
user: login: damccorm
body: This block (until the break) is fairly tough to follow because its split from context - would it make sense to move some of this logic into `_YamlTestCase` or (probably preferred) `yaml_testing`? That way it is closer to the context  of how we can expect this to be structured.

An alternate way of cleaning this up would make a function to retrieve `test.fixes.items()` which could come with a clear doc of what it is returning.
created_at: 2025-04-08T14:57:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,292 @@
+---
+type: languages
+title: "Apache Beam YAML Testing"
+---
+<!--
+    Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+-->
+
+# Beam YAML Tests
+
+A robust testing story is an important part of any production setup.
+Though the various built-in (and externally provided) transform in a Beam YAML
+pipeline can be expected to be well tested, it can be important to have tests
+that ensure the pipeline as a whole behaves as expected.  This is particularly
+true for transforms that contain non-trivial UDF logic.
+
+# Whole pipeline tests
+
+For example, consider the example word count pipeline.
+
+```
+pipeline:
+  transforms:
+  - type: ReadFromText
+    name: Read from GCS
+    config:
+      path: gs://dataflow-samples/shakespeare/kinglear.txt
+  - type: MapToFields
+    name: Split words
+    config:
+      language: python
+      fields:
+        word:
+          callable: |
+            import re
+            def all_words(row):
+              return re.findall(r'[a-z]+', row.line.lower())
+        value: 1
+    input: Read from GCS
+  - type: Explode
+    name: Explode word arrays
+    config:
+      fields: [word]
+    input: Split words
+  - type: Combine
+    name: Count words
+    config:
+      group_by: [word]
+      combine:
+        value: sum
+    input: Explode word arrays
+  - type: MapToFields
+    name: Format output
+    config:
+      language: python
+      fields:
+        output: "word + ': ' + str(value)"
+    input: Count words
+  - type: WriteToText
+    name: Write to GCS
+    config:
+      path: gs://bucket/counts.txt
+    input: Format output
+
+tests: []
+```
+
+To write tests for this pipeline, one creates a `tests` section that enumerates
+a number of tests, each of which provide example input and assert the expected
+output is produced.  An example test might be as follows
+
+```
+tests:
+- name: MyRegressionTest
+  mock_outputs:
+    - name: Read from GCS
+      elements:
+        - line: "Nothing can come of nothing"
+  expected_inputs:
+    - name: Write to GCS
+      elements:
+        - output: 'nothing: 2'
+        - output: 'can: 1'
+        - output: 'come: 1'
+        - output: 'of: 1'
+```
+
+The `mock_outputs` section designates that the transform named `Read from GCS`
+should produce the single row `{line: "Nothing can come of nothing"}` for the
+purposes of this test, and the `expected_inputs` section indicates that the
+transform `Write to GCS` should expect to receive exactly the given elements.
+Neither the actual Read transform nor Write transform from the original
+pipelines are executed when running the test, but all intermediate transforms
+are.
+
+This test can then be executed by running
+
+```
+python -m apache_beam.yaml.main \
+    --yaml_pipeline_file=wordcount.yaml \
+    --tests
+```
+
+Alternatively, the a `tests:` block may be placed in a separate file and be
+validated by running
+
+```
+python -m apache_beam.yaml.main \
+    --yaml_pipeline_file=wordcount.yaml \
+    --tests \
+    --test_suite=test_file.yaml
+```
+
+Neither the actual Read transform nor Write transform from the original
+pipelines are executed when running the test, but all intermediate transforms
+are.  For hermeticity, we require that all inputs (with the exception of
+`Create` that are needed to compute the expected outputs are explicitly mocked;
+to explicitly allow a sources to be executed as part of a test their names or
+types can be enumerated in an `allowed_sources` attribute of the test
+specification.
+
+
+## Pipeline fragment tests
+
+One can also tests a portion of a pipeline using the `mock_inputs` and
+`expected_outputs` section of a test, for example
+
+```
+tests:
+- name: TestSplittingWithPunctuation
+  mock_inputs:
+    - name: Split words
+      elements:
+        - line: "lots-of-words"
+        - line: "...and more"
+  expected_outputs:
+    - name: Explode
+      elements:
+        - word: lots
+          value: 1
+        - word: of
+          value: 1
+        - word: words
+          value: 1
+        - word: and
+          value: 1
+        - word: more
+          value: 1
+
+- name: TestCombineAndFormat
+  mock_inputs:
+    - name: Count words
+      elements:
+        - word: more
+          value: 1
+        - word: and
+          value: 1
+        - word: more
+          value: 1
+  expected_outputs:
+    - name: Format output
+      elements:
+        - output: "more: 2"
+        - output: "and: 1"
+```
+
+As before, each test only executes the portion of the pipeline between the
+mock inputs and expected outputs.  Note that the named transform in a
+`mock_inputs` specification *is* executed, while the named transform of a
+`mock_oupputs` specification is not.
+Similarly, the named transform of a `expected_inputs` specification is *not*
+executed, while the named transform of an `expected_outputs` necessarily is.
+
+
+## Automatically generating tests.
+
+In an effort to make tests as easy to write and maintain as possible,
+Beam YAML provides utilities to compute the expected outputs for your tests.
+Running
+
+```
+python -m apache_beam.yaml.main \
+    --yaml_pipeline_file=wordcount.yaml \
+    --tests \
+    [--test_suite=...] \
+    --fix_tests
+```
+
+will update any existing `expected_input` and `expected_output` blocks of your
+pipeline to contain the actual values computed during the test.
+(Of course, it is on any user of this flag to verify that the produced values
+are as expected.)
+This can be useful in authoring tests as well--one can simply specify a
+nonsensical or empty elements block in the expectation and the `--fix_tests`
+flag will populate it for you.
+
+Beam YAML also has a `--create_test` flag which can be used to create an
path: website/www/site/content/en/documentation/sdks/yaml-testing.md
user: login: damccorm
body: I think it would actually be great to lead with this in this section. I expect that most users would use this to get a suite set up, then they could add more tests.

It is also worth mentioning that you need a test-suite flag when using this.
created_at: 2025-04-08T15:09:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -90,6 +95,26 @@ def _parse_arguments(argv):
       type=json.loads,
       help='A json dict of variables used when invoking the jinja preprocessor '
       'on the provided yaml pipeline.')
+  parser.add_argument(
+      '--test',
+      action=argparse.BooleanOptionalAction,
+      help='Run the tests associated with the given pipeline, rather than the '
+      'pipeline itself.')
+  parser.add_argument(
+      '--fix_tests',
+      action=argparse.BooleanOptionalAction,
+      help='Update failing test expectations to match the actual ouput.')
path: sdks/python/apache_beam/yaml/main.py
user: login: robertwb
body: This only requires test-suite if the pipeline is templated. I'll update the doc. 
created_at: 2025-04-08T20:32:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -90,6 +95,26 @@ def _parse_arguments(argv):
       type=json.loads,
       help='A json dict of variables used when invoking the jinja preprocessor '
       'on the provided yaml pipeline.')
+  parser.add_argument(
+      '--test',
+      action=argparse.BooleanOptionalAction,
+      help='Run the tests associated with the given pipeline, rather than the '
+      'pipeline itself.')
+  parser.add_argument(
+      '--fix_tests',
+      action=argparse.BooleanOptionalAction,
+      help='Update failing test expectations to match the actual ouput.')
path: sdks/python/apache_beam/yaml/main.py
user: login: robertwb
body: Same.
created_at: 2025-04-08T20:33:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -130,12 +155,112 @@ def run(argv=None):
       print('Running pipeline...')
 
 
-def build_pipeline_components_from_argv(argv):
+def run_tests(argv=None, exit=True):
+  known_args, pipeline_args, _, pipeline_yaml = _build_pipeline_yaml_from_argv(
+      argv)
+  pipeline_spec = yaml.load(pipeline_yaml, Loader=yaml_transform.SafeLineLoader)
+  options = _build_pipeline_options(pipeline_spec, pipeline_args)
+
+  if known_args.create_test and not known_args.fix_tests:
+    result = unittest.TestResult()
+    tests = []
+  else:
+    if known_args.test_suite:
+      with open(known_args.test_suite) as fin:
+        test_suite = yaml.load(fin, Loader=yaml_transform.SafeLineLoader) or {}
+      if 'tests' not in test_suite or not isinstance(test_suite['tests'], list):
+        raise TypeError('tests attribute must be a list of test specifications')
+      test_specs = test_suite['tests']
+    else:
+      test_specs = pipeline_spec.get('tests', [])
+      if not isinstance(test_specs, list):
+        raise TypeError('tests attribute must be a list of test specifications')
+    if not test_specs:
+      raise RuntimeError('No tests found.')
+
+    with _fix_xlang_instant_coding():
+      tests = [
+          _YamlTestCase(
+              pipeline_spec, test_spec, options, known_args.fix_tests)
+          for test_spec in test_specs
+      ]
+      suite = unittest.TestSuite(tests)
+      result = unittest.TextTestRunner().run(suite)
+
+  if known_args.fix_tests or known_args.create_test:
+    update_tests(known_args, pipeline_spec, options, tests)
+
+  if exit:
+    # emulates unittest.main()
+    sys.exit(0 if result.wasSuccessful() else 1)
+  else:
+    if not result.wasSuccessful():
+      raise RuntimeError(result)
+
+
+def update_tests(known_args, pipeline_spec, options, tests):
+  if known_args.test_suite:
+    path = known_args.test_suite
+    if not os.path.exists(path) and known_args.create_test:
+      with open(path, 'w') as fout:
+        fout.write('tests: []')
+  elif known_args.yaml_pipeline_file:
+    path = known_args.yaml_pipeline_file
+  else:
+    raise RuntimeError(
+        'Test fixing only supported for file-backed tests. '
+        'Please use the --test_suite flag.')
+  with open(path) as fin:
+    original_yaml = fin.read()
+  if path == known_args.yaml_pipeline_file and pipeline_yaml.strip(
+  ) != original_yaml.strip():
+    raise RuntimeError(
+        'In-file test fixing not yet supported for templated pipelines. '
+        'Please use the --test_suite flag.')
+  updated_spec = yaml.load(original_yaml, Loader=yaml.SafeLoader) or {}
+
+  if known_args.fix_tests:
+    for ix, test in enumerate(tests):
+      if test.fixes:
path: sdks/python/apache_beam/yaml/main.py
user: login: robertwb
body: create_test uses this path as well
created_at: 2025-04-08T23:03:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -130,12 +155,112 @@ def run(argv=None):
       print('Running pipeline...')
 
 
-def build_pipeline_components_from_argv(argv):
+def run_tests(argv=None, exit=True):
+  known_args, pipeline_args, _, pipeline_yaml = _build_pipeline_yaml_from_argv(
+      argv)
+  pipeline_spec = yaml.load(pipeline_yaml, Loader=yaml_transform.SafeLineLoader)
+  options = _build_pipeline_options(pipeline_spec, pipeline_args)
+
+  if known_args.create_test and not known_args.fix_tests:
path: sdks/python/apache_beam/yaml/main.py
user: login: robertwb
body: Sure. Done.
created_at: 2025-04-09T00:45:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -130,12 +155,112 @@ def run(argv=None):
       print('Running pipeline...')
 
 
-def build_pipeline_components_from_argv(argv):
+def run_tests(argv=None, exit=True):
+  known_args, pipeline_args, _, pipeline_yaml = _build_pipeline_yaml_from_argv(
+      argv)
+  pipeline_spec = yaml.load(pipeline_yaml, Loader=yaml_transform.SafeLineLoader)
+  options = _build_pipeline_options(pipeline_spec, pipeline_args)
+
+  if known_args.create_test and not known_args.fix_tests:
+    result = unittest.TestResult()
+    tests = []
+  else:
+    if known_args.test_suite:
+      with open(known_args.test_suite) as fin:
+        test_suite = yaml.load(fin, Loader=yaml_transform.SafeLineLoader) or {}
+      if 'tests' not in test_suite or not isinstance(test_suite['tests'], list):
+        raise TypeError('tests attribute must be a list of test specifications')
path: sdks/python/apache_beam/yaml/main.py
user: login: robertwb
body: Updated and simplified.
created_at: 2025-04-09T00:45:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -90,6 +95,26 @@ def _parse_arguments(argv):
       type=json.loads,
       help='A json dict of variables used when invoking the jinja preprocessor '
       'on the provided yaml pipeline.')
+  parser.add_argument(
+      '--test',
+      action=argparse.BooleanOptionalAction,
+      help='Run the tests associated with the given pipeline, rather than the '
+      'pipeline itself.')
+  parser.add_argument(
+      '--fix_tests',
+      action=argparse.BooleanOptionalAction,
+      help='Update failing test expectations to match the actual ouput.')
path: sdks/python/apache_beam/yaml/main.py
user: login: robertwb
body: Done.
created_at: 2025-04-09T00:46:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -90,6 +95,26 @@ def _parse_arguments(argv):
       type=json.loads,
       help='A json dict of variables used when invoking the jinja preprocessor '
       'on the provided yaml pipeline.')
+  parser.add_argument(
+      '--test',
+      action=argparse.BooleanOptionalAction,
+      help='Run the tests associated with the given pipeline, rather than the '
+      'pipeline itself.')
+  parser.add_argument(
+      '--fix_tests',
+      action=argparse.BooleanOptionalAction,
+      help='Update failing test expectations to match the actual ouput.')
path: sdks/python/apache_beam/yaml/main.py
user: login: robertwb
body: Done.
created_at: 2025-04-09T00:47:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -130,12 +155,112 @@ def run(argv=None):
       print('Running pipeline...')
 
 
-def build_pipeline_components_from_argv(argv):
+def run_tests(argv=None, exit=True):
+  known_args, pipeline_args, _, pipeline_yaml = _build_pipeline_yaml_from_argv(
+      argv)
+  pipeline_spec = yaml.load(pipeline_yaml, Loader=yaml_transform.SafeLineLoader)
+  options = _build_pipeline_options(pipeline_spec, pipeline_args)
+
+  if known_args.create_test and not known_args.fix_tests:
+    result = unittest.TestResult()
+    tests = []
+  else:
+    if known_args.test_suite:
+      with open(known_args.test_suite) as fin:
+        test_suite = yaml.load(fin, Loader=yaml_transform.SafeLineLoader) or {}
+      if 'tests' not in test_suite or not isinstance(test_suite['tests'], list):
+        raise TypeError('tests attribute must be a list of test specifications')
+      test_specs = test_suite['tests']
+    else:
+      test_specs = pipeline_spec.get('tests', [])
+      if not isinstance(test_specs, list):
+        raise TypeError('tests attribute must be a list of test specifications')
+    if not test_specs:
+      raise RuntimeError('No tests found.')
+
+    with _fix_xlang_instant_coding():
+      tests = [
+          _YamlTestCase(
+              pipeline_spec, test_spec, options, known_args.fix_tests)
+          for test_spec in test_specs
+      ]
+      suite = unittest.TestSuite(tests)
+      result = unittest.TextTestRunner().run(suite)
+
+  if known_args.fix_tests or known_args.create_test:
+    update_tests(known_args, pipeline_spec, options, tests)
+
+  if exit:
+    # emulates unittest.main()
+    sys.exit(0 if result.wasSuccessful() else 1)
+  else:
+    if not result.wasSuccessful():
+      raise RuntimeError(result)
+
+
+def update_tests(known_args, pipeline_spec, options, tests):
+  if known_args.test_suite:
+    path = known_args.test_suite
+    if not os.path.exists(path) and known_args.create_test:
+      with open(path, 'w') as fout:
+        fout.write('tests: []')
+  elif known_args.yaml_pipeline_file:
+    path = known_args.yaml_pipeline_file
+  else:
+    raise RuntimeError(
+        'Test fixing only supported for file-backed tests. '
+        'Please use the --test_suite flag.')
+  with open(path) as fin:
+    original_yaml = fin.read()
+  if path == known_args.yaml_pipeline_file and pipeline_yaml.strip(
+  ) != original_yaml.strip():
+    raise RuntimeError(
+        'In-file test fixing not yet supported for templated pipelines. '
+        'Please use the --test_suite flag.')
+  updated_spec = yaml.load(original_yaml, Loader=yaml.SafeLoader) or {}
+
+  if known_args.fix_tests:
+    for ix, test in enumerate(tests):
+      if test.fixes:
path: sdks/python/apache_beam/yaml/main.py
user: login: robertwb
body: Good call. I've cleaned this up (and moved it all into yaml_testing).
created_at: 2025-04-09T01:10:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,292 @@
+---
+type: languages
+title: "Apache Beam YAML Testing"
+---
+<!--
+    Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+-->
+
+# Beam YAML Tests
+
+A robust testing story is an important part of any production setup.
+Though the various built-in (and externally provided) transform in a Beam YAML
+pipeline can be expected to be well tested, it can be important to have tests
+that ensure the pipeline as a whole behaves as expected.  This is particularly
+true for transforms that contain non-trivial UDF logic.
+
+# Whole pipeline tests
+
+For example, consider the example word count pipeline.
+
+```
+pipeline:
+  transforms:
+  - type: ReadFromText
+    name: Read from GCS
+    config:
+      path: gs://dataflow-samples/shakespeare/kinglear.txt
+  - type: MapToFields
+    name: Split words
+    config:
+      language: python
+      fields:
+        word:
+          callable: |
+            import re
+            def all_words(row):
+              return re.findall(r'[a-z]+', row.line.lower())
+        value: 1
+    input: Read from GCS
+  - type: Explode
+    name: Explode word arrays
+    config:
+      fields: [word]
+    input: Split words
+  - type: Combine
+    name: Count words
+    config:
+      group_by: [word]
+      combine:
+        value: sum
+    input: Explode word arrays
+  - type: MapToFields
+    name: Format output
+    config:
+      language: python
+      fields:
+        output: "word + ': ' + str(value)"
+    input: Count words
+  - type: WriteToText
+    name: Write to GCS
+    config:
+      path: gs://bucket/counts.txt
+    input: Format output
+
+tests: []
+```
+
+To write tests for this pipeline, one creates a `tests` section that enumerates
+a number of tests, each of which provide example input and assert the expected
+output is produced.  An example test might be as follows
+
+```
+tests:
+- name: MyRegressionTest
+  mock_outputs:
+    - name: Read from GCS
+      elements:
+        - line: "Nothing can come of nothing"
+  expected_inputs:
+    - name: Write to GCS
+      elements:
+        - output: 'nothing: 2'
+        - output: 'can: 1'
+        - output: 'come: 1'
+        - output: 'of: 1'
+```
+
+The `mock_outputs` section designates that the transform named `Read from GCS`
+should produce the single row `{line: "Nothing can come of nothing"}` for the
+purposes of this test, and the `expected_inputs` section indicates that the
+transform `Write to GCS` should expect to receive exactly the given elements.
+Neither the actual Read transform nor Write transform from the original
+pipelines are executed when running the test, but all intermediate transforms
+are.
+
+This test can then be executed by running
+
+```
+python -m apache_beam.yaml.main \
+    --yaml_pipeline_file=wordcount.yaml \
+    --tests
+```
+
+Alternatively, the a `tests:` block may be placed in a separate file and be
+validated by running
+
+```
+python -m apache_beam.yaml.main \
+    --yaml_pipeline_file=wordcount.yaml \
+    --tests \
+    --test_suite=test_file.yaml
+```
+
+Neither the actual Read transform nor Write transform from the original
+pipelines are executed when running the test, but all intermediate transforms
+are.  For hermeticity, we require that all inputs (with the exception of
+`Create` that are needed to compute the expected outputs are explicitly mocked;
+to explicitly allow a sources to be executed as part of a test their names or
+types can be enumerated in an `allowed_sources` attribute of the test
+specification.
+
+
+## Pipeline fragment tests
+
+One can also tests a portion of a pipeline using the `mock_inputs` and
+`expected_outputs` section of a test, for example
+
+```
+tests:
+- name: TestSplittingWithPunctuation
+  mock_inputs:
+    - name: Split words
+      elements:
+        - line: "lots-of-words"
+        - line: "...and more"
+  expected_outputs:
+    - name: Explode
+      elements:
+        - word: lots
+          value: 1
+        - word: of
+          value: 1
+        - word: words
+          value: 1
+        - word: and
+          value: 1
+        - word: more
+          value: 1
+
+- name: TestCombineAndFormat
+  mock_inputs:
+    - name: Count words
+      elements:
+        - word: more
+          value: 1
+        - word: and
+          value: 1
+        - word: more
+          value: 1
+  expected_outputs:
+    - name: Format output
+      elements:
+        - output: "more: 2"
+        - output: "and: 1"
+```
+
+As before, each test only executes the portion of the pipeline between the
+mock inputs and expected outputs.  Note that the named transform in a
+`mock_inputs` specification *is* executed, while the named transform of a
+`mock_oupputs` specification is not.
+Similarly, the named transform of a `expected_inputs` specification is *not*
+executed, while the named transform of an `expected_outputs` necessarily is.
+
+
+## Automatically generating tests.
+
+In an effort to make tests as easy to write and maintain as possible,
+Beam YAML provides utilities to compute the expected outputs for your tests.
+Running
+
+```
+python -m apache_beam.yaml.main \
+    --yaml_pipeline_file=wordcount.yaml \
+    --tests \
+    [--test_suite=...] \
+    --fix_tests
+```
+
+will update any existing `expected_input` and `expected_output` blocks of your
+pipeline to contain the actual values computed during the test.
+(Of course, it is on any user of this flag to verify that the produced values
+are as expected.)
+This can be useful in authoring tests as well--one can simply specify a
+nonsensical or empty elements block in the expectation and the `--fix_tests`
+flag will populate it for you.
+
+Beam YAML also has a `--create_test` flag which can be used to create an
path: website/www/site/content/en/documentation/sdks/yaml-testing.md
user: login: robertwb
body: Done.
created_at: 2025-04-09T01:13:37Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-04-08T00:27:03Z
body: Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:

R: @tvalentyn for label python.
R: @damccorm for label website.

Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).
author_association: CONTRIBUTOR
user: login: damccorm
created_at: 2025-04-09T14:26:59Z
body: > Thanks for looking at this! Given your comments are throughout the files, should I assume you're reviewing the entire change? (I tried to break it up into pull reviews both to get it out there sooner and spread the load, but maybe that's not to helpful after all.)

Yeah, I did review the whole thing - I reviewed before I realized you had split it up, thanks for doing that though. At this point, I'd probably just vote for us to take this forward
author_association: CONTRIBUTOR