Repository: microsoft/promptflow

Issue:
[BUG] Streaming requires Generator where Iterator should suffice
**Describe the bug**
For the user code to provide streaming content, instead of the more generic Iterator protocol, Promptflow relies on the more specific `GeneratorType`. That is requiring the user to unnecessarily wrap their iterators in generators to stream content in more generic scenarios (such as streaming langchain results). Is there a reason PF depends on GeneratorType instead of the iterator protocol? Also, see this note in the [python source](https://github.com/python/cpython/blob/f65497fd252a4a4df960da04d68e8316b58624c0/Lib/types.py#L6-L10):
```
# Iterators in Python aren't a matter of type but of protocol.  A large
# and changing number of builtin types implement *some* flavor of
# iterator.  Don't check the type!  Use hasattr to check for both
# "__iter__" and "__next__" attributes instead.
```

**Concrete Issue:** 
When returning an iterator from my tool to enable streaming, I get the following error when running `pf flow test`
```
Flow test failed with UserErrorException: Exception: The output 'answer' for flow is incorrect. The output value is 
not JSON serializable. JSON dump failed: (TypeError) Object of type WordIterator is not JSON serializable. Please 
verify your flow output and make sure the value serializable.
```

**How To Reproduce the bug**
Here is my flow.dag.yaml:
```yaml
id: template_chat_flow
name: Template Chat Flow
inputs:
  chat_history:
    type: list
    default: []
  question:
    type: string
    default: What is the meaning of life?
outputs:
  answer:
    type: string
    is_chat_output: true
    reference: ${stream.output}
nodes:
- name: stream
  type: python
  source:
    type: code
    path: stream.py
  inputs:
    input: ${inputs.question}
```

And here is my stream.py
```python
from promptflow import tool

class WordIterator:
    def __init__(self, input: str):
        self.input = input

    def __iter__(self):
        return self

    def __next__(self):
        if self.input:
            word, *rest = self.input.split(" ")
            self.input = " ".join(rest)
            return f"{word} "
        else:
            raise StopIteration  

@tool
def my_python_tool(input: str):
    iterator = WordIterator(input)
    assert hasattr(iterator, "__iter__")
    assert hasattr(iterator, "__next__")
    return iterator
```
With the above PF run `pf flow test --flow .` -- I get this error:
```
2023-09-19 11:01:17 +0200   42558 execution          INFO     Start to run 1 nodes with concurrency level 16.
2023-09-19 11:01:17 +0200   42558 execution.flow     INFO     Executing node stream. node run id: c4ddaddd-6a38-44fd-9ab1-3258fb88bb37_stream_0
2023-09-19 11:01:17 +0200   42558 execution.flow     WARNING  Output of stream is not json serializable, use str to store it.
2023-09-19 11:01:17 +0200   42558 execution.flow     INFO     Node stream completes.
Flow test failed with UserErrorException: Exception: The output 'answer' for flow is incorrect. The output value is not JSON serializable. JSON dump failed: (TypeError) Object of type WordIterator is not JSON serializable. Please verify your flow output and make sure the value serializable.
```

**Expected behavior**
It should read out the iterator and stream the chunks to the caller -- `pf flow test --flow .` should look like this:
```
2023-09-19 10:34:05 +0200   40375 execution          INFO     Start to run 1 nodes with concurrency level 16.
2023-09-19 10:34:05 +0200   40375 execution.flow     INFO     Executing node stream. node run id: 24e60c4d-606a-4fc5-8e4c-cc4a5c41d6c8_stream_0
2023-09-19 10:34:05 +0200   40375 execution.flow     WARNING  Output of stream is not json serializable, use str to store it.
2023-09-19 10:34:05 +0200   40375 execution.flow     INFO     Node stream completes.
{
    "answer": "What is the meaning of life? "
}
```

**Running Information(please complete the following information):**
 - Promptflow Package Version using `pf -v`: 0.1.0b6
 - Operating System: 
```
ProductName:            macOS
ProductVersion:         13.5.2
BuildVersion:           22G91
```
 - Python Version using `python --version`: Python 3.10.12

**Additional context**
If I wrap the iterator in a generator, everything works as expected:

```python
from promptflow import tool

class WordIterator:
    def __init__(self, input: str):
        self.input = input

    def __iter__(self):
        return self

    def __next__(self):
        if self.input:
            word, *rest = self.input.split(" ")
            self.input = " ".join(rest)
            return f"{word} "
        else:
            raise StopIteration  
    
    def to_generator(self):
        try:
            while True:
                yield next(self)
        except StopIteration:
            pass

@tool
def my_python_tool(input: str):
    iterator = WordIterator(input).to_generator()
    assert hasattr(iterator, "__iter__")
    assert hasattr(iterator, "__next__")
    return iterator
```



Discussion:

Resolving patch:
diff --git a/src/promptflow/promptflow/_core/tracer.py b/src/promptflow/promptflow/_core/tracer.py
--- a/src/promptflow/promptflow/_core/tracer.py
+++ b/src/promptflow/promptflow/_core/tracer.py
@@ -5,9 +5,9 @@
 import inspect
 import json
 import logging
+from collections.abc import Iterator
 from contextvars import ContextVar
 from datetime import datetime
-from types import GeneratorType
 from typing import Optional
 
 from promptflow._core.generator_proxy import GeneratorProxy, generate_from_proxy
@@ -106,7 +106,7 @@ def pop(cls, output=None, error: Optional[Exception] = None):
 
     def _pop(self, output=None, error: Optional[Exception] = None):
         last_trace = self._trace_stack[-1]
-        if isinstance(output, GeneratorType):
+        if isinstance(output, Iterator):
             output = GeneratorProxy(output)
         if output is not None:
             last_trace.output = self.to_serializable(output)


Test patch:
diff --git a/src/promptflow/tests/executor/unittests/_core/test_generator_proxy.py b/src/promptflow/tests/executor/unittests/_core/test_generator_proxy.py
--- a/src/promptflow/tests/executor/unittests/_core/test_generator_proxy.py
+++ b/src/promptflow/tests/executor/unittests/_core/test_generator_proxy.py
@@ -8,6 +8,10 @@ def generator():
         yield i
 
 
+def iterator():
+    return iter([0, 1, 2])
+
+
 @pytest.mark.unittest
 def test_generator_proxy_next():
     proxy = GeneratorProxy(generator())
@@ -42,3 +46,39 @@ def test_generate_from_proxy():
         assert i == next(original_generator)
 
     assert proxy.items == [0, 1, 2]
+
+
+@pytest.mark.unittest
+def test_iterator_proxy_next():
+    proxy = GeneratorProxy(iterator())
+    assert proxy.items == []
+    assert next(proxy) == 0
+    assert next(proxy) == 1
+    assert next(proxy) == 2
+
+    with pytest.raises(StopIteration):
+        next(proxy)
+
+    assert proxy.items == [0, 1, 2]
+
+
+@pytest.mark.unittest
+def test_iterator_proxy_iter():
+    original_iterator = iterator()
+    proxy = GeneratorProxy(iterator())
+
+    for num in proxy:
+        assert num == next(original_iterator)
+
+    assert proxy.items == [0, 1, 2]
+
+
+@pytest.mark.unittest
+def test_generate_from_iterator_proxy():
+    proxy = GeneratorProxy(iterator())
+    original_iterator = iterator()
+
+    for i in generate_from_proxy(proxy):
+        assert i == next(original_iterator)
+
+    assert proxy.items == [0, 1, 2]