opened
mat-hek
username_0
Testing API is now quite confusing. Here are some reasons:
- It's difficult to determine from the name and arguments how does `assert_receive_message` differ from usual `assert_receive`
- Checking for something as simple and common as End Of Stream requires subscribing on `handle_notification`
- Messages from testing elements are received with `assert_receive`, while messages passed through the pipeline - with `assert_receive_message`
- There is no way to distinguish pipeline or element that sent a message if there are many of them

Suggested changes are:
- Make all testing elements send notifications to pipeline instead of messages to the testing process
- Always implicitly subscribe on `handle_notification`
- Instead of `assert_receive_message`, make separate assertions like below:
  - `assert_pipeline_notification(pipeline_pid, element_name, message)`
  - `assert_pipeline_playback_changed(pipeline_pid, previous_playback_state, current_playback_state)`
  - `assert_pipeline_handle_other(pipeline_pid, message)`
Improve Membrane.Testing API
issue
created
Hajto
username_1
To be discussed on meeting:

# New Membrane Testing API

## Pipeline

By default Pipeline will be subscribed to all callbacks. If someone explicitly
subscribes to set of `callbacks`, then only those will be available for assertion.

`assert_receive_message` will be removed in favor of set of specialized assertions.
```elixir
defmodule Membrane.Testing.Assertions do
  # Old split into different assertions
  def assert_pipeline_notified(pipeline_pid, element_name, message)
  def assert_pipeline_playback_changed(pipeline_pid, previous_playback_state, current_playback_state)
  def assert_pipeline_received(pipeline_pid, message)
end
```

NOTE: Pipeline will receive messages from elements as notifications, they will
have to be distinguishable from those that are expected to flow through the
pipeline.

## Testing Sink

`Membrane.Testing.Sink` will no longer sent messages to test_process. It will
notify pipeline about it and pipeline will be responsible for executing assertions.

```elixir
defmodule Membrane.Testing.Sink do
  # We shall add new quality of life assertions for most common use cases

  # I don't see use case for start_of_stream,
  # but implementing only end_of_stream
  def assert_start_of_stream(pipeline_pid, sink_name) 
  def assert_end_of_stream(pipeline_pid, sink_name)
  def assert_event(pipeline_pid, sink_name, event_pattern)
  def assert_buffer(pipeline_pid, sink_name, buffer_pattern)
end
```

Sink name shall be added to enable assertion on different named Sinks so that
pipelines with multiple outputs can be supported.

## Testing Source

### Plan A: merging sources

`DataSource` shall be removed in favor of adding helper function to `Source`,
which streams takes data as argument and returns generator function.

Generator function shall supply demands and when supply runs out it shall send
end of stream event.

Data source can (not necessarily must) defined as follows:
```elixir
def data_source(data, bufferizer \\ identity) do
  actions = Enum.map(data, fn element -> {:buffer, {:output, bufferizer.(element)}} end)

  fn cnt, size ->
    if(cnt != 0, do: cnt, else: actions)
    |> Enum.split(size)
    ~>> ({to_send, []} -> {to_send ++ [{:event, {:output, %EndOfStream{}}}], []})
  end
end
```

then it could be potentially used as follows:

```elixir
%Options{
  elements: [
    a_source: %Testing.Source{generator: data_source([1,2,3])}
  ]
}
```

### Plan B: both sources stay separated

If `DataSource` will not incorporated into `Testing.Source`, then there should
be either added option to accept list of Buffers as data, because there are
cases when appropriate Metadata must be set (for example H264 depayloader
which requires RTP Sequence Number to be present).

## Inspector Filter

This is only **proposal**.

Inspector is `filter` type element which copies all the buffers it receives and
forwards them to pipeline so there can be made assertion on them. Such element
potentially can be used for creating more detailed assertion on integration
pipelines and help locating error more easily.

## Example test after all the changes

```elixir
test "I am very important test and I MUST always pass" do
  alias Membrane.Testing
  {:ok, pipeline} = Testing.Pipeline.start_link(%Testing.Pipeline.Options{
    elements: [
      source: %Testing.Source{generator: data_source(1..100)},
      rtp_parser: RTP,
      inspektor: Inspector,
      depayloader: Depayloader,
      sink: %Testing.Sink{}
    ]
  })

  Membrane.Pipeline.play(pipeline)

  assert_end_of_stream pipeline, :sink
  Enum.each(rtp_parsed, fn element -> 
    assert_buffer pipeline, :inspektor, ^element 
  end)
  Enum.each(depayloaded, fn element ->
    assert_buffer pipeline, :inspektor, ^element 
  end)
end
```
comment
closed
mat-hek
username_0
issue