Repository: php-llm/llm-chain
Pull request #270: feat: Add Raw Response Metadata Handling - GPT Token Usage Example
Author: login: DZunke
Labels: name: feature
Created: 2025-04-08T11:12:21Z | Merged: 2025-04-25T08:38:46Z

Currently it is not really possible to get into details of the interaction between user and model because the HTTPResponse is hidden behind the response converter and is then forgotten forever. So the only possibility currently to get details from the interaction would be a custom response and a custom response converter. 

As a draft and to collect your ideas and other input i want to throw this stone into the lake with having the HTTPResponse also transported into the direction of the output processors. Could this give a bit more flexibility? What do you think? 

Please have in mind this is only a draft to discuss. I thought about having this also possible if a request fails with an error status code in the future so the output processors could handle in a separation of convern way if there should be an exception thrown. Or do we maybe need an additional layer on top of the response converters itself? 

For sure this is the simplest use case with the text response in mind. Maybe for the streamed response it would need a more "recording" approach with an output processor. But would surely be possible. 


Diff:
diff --git a/examples/chat-gpt-openai-metadata.php b/examples/chat-gpt-openai-metadata.php
new file mode 100644
index 00000000..2901104e
--- /dev/null
+++ b/examples/chat-gpt-openai-metadata.php
@@ -0,0 +1,38 @@
+<?php
+
+use PhpLlm\LlmChain\Bridge\OpenAI\GPT;
+use PhpLlm\LlmChain\Bridge\OpenAI\PlatformFactory;
+use PhpLlm\LlmChain\Bridge\OpenAI\TokenOutputProcessor;
+use PhpLlm\LlmChain\Chain;
+use PhpLlm\LlmChain\Model\Message\Message;
+use PhpLlm\LlmChain\Model\Message\MessageBag;
+use Symfony\Component\Dotenv\Dotenv;
+
+require_once dirname(__DIR__).'/vendor/autoload.php';
+(new Dotenv())->loadEnv(dirname(__DIR__).'/.env');
+
+if (empty($_ENV['OPENAI_API_KEY'])) {
+    echo 'Please set the OPENAI_API_KEY environment variable.'.PHP_EOL;
+    exit(1);
+}
+
+$platform = PlatformFactory::create($_ENV['OPENAI_API_KEY']);
+$llm = new GPT(GPT::GPT_4O_MINI, [
+    'temperature' => 0.5, // default options for the model
+]);
+
+$chain = new Chain($platform, $llm, outputProcessors: [new TokenOutputProcessor()]);
+$messages = new MessageBag(
+    Message::forSystem('You are a pirate and you write funny.'),
+    Message::ofUser('What is the Symfony framework?'),
+);
+$response = $chain->call($messages, [
+    'max_tokens' => 500, // specific options just for this call
+]);
+
+$metadata = $response->getMetadata();
+
+echo 'Utilized Tokens: '.$metadata['total_tokens'].PHP_EOL;
+echo '-- Prompt Tokens: '.$metadata['prompt_tokens'].PHP_EOL;
+echo '-- Completion Tokens: '.$metadata['completion_tokens'].PHP_EOL;
+echo 'Remaining Tokens: '.$metadata['remaining_tokens'].PHP_EOL;
diff --git a/src/Bridge/OpenAI/DallE/ImageResponse.php b/src/Bridge/OpenAI/DallE/ImageResponse.php
index 58c8ffc1..eabece9d 100644
--- a/src/Bridge/OpenAI/DallE/ImageResponse.php
+++ b/src/Bridge/OpenAI/DallE/ImageResponse.php
@@ -4,9 +4,9 @@
 
 namespace PhpLlm\LlmChain\Bridge\OpenAI\DallE;
 
-use PhpLlm\LlmChain\Model\Response\ResponseInterface;
+use PhpLlm\LlmChain\Model\Response\BaseResponse;
 
-class ImageResponse implements ResponseInterface
+class ImageResponse extends BaseResponse
 {
     /** @var list<Base64Image|UrlImage> */
     private readonly array $images;
diff --git a/src/Bridge/OpenAI/TokenOutputProcessor.php b/src/Bridge/OpenAI/TokenOutputProcessor.php
new file mode 100644
index 00000000..69089b5b
--- /dev/null
+++ b/src/Bridge/OpenAI/TokenOutputProcessor.php
@@ -0,0 +1,42 @@
+<?php
+
+declare(strict_types=1);
+
+namespace PhpLlm\LlmChain\Bridge\OpenAI;
+
+use PhpLlm\LlmChain\Chain\Output;
+use PhpLlm\LlmChain\Chain\OutputProcessor;
+use PhpLlm\LlmChain\Model\Response\StreamResponse;
+
+final class TokenOutputProcessor implements OutputProcessor
+{
+    public function processOutput(Output $output): void
+    {
+        if ($output->response instanceof StreamResponse) {
+            // Streams have to be handled manually as the tokens are part of the streamed chunks
+            return;
+        }
+
+        $rawResponse = $output->response->getRawResponse();
+        if (null === $rawResponse) {
+            return;
+        }
+
+        $metadata = $output->response->getMetadata();
+
+        $metadata->add(
+            'remaining_tokens',
+            (int) $rawResponse->getHeaders(false)['x-ratelimit-remaining-tokens'][0],
+        );
+
+        $content = $rawResponse->toArray(false);
+
+        if (!\array_key_exists('usage', $content)) {
+            return;
+        }
+
+        $metadata->add('prompt_tokens', $content['usage']['prompt_tokens'] ?? null);
+        $metadata->add('completion_tokens', $content['usage']['completion_tokens'] ?? null);
+        $metadata->add('total_tokens', $content['usage']['total_tokens'] ?? null);
+    }
+}
diff --git a/src/Chain/Toolbox/StreamResponse.php b/src/Chain/Toolbox/StreamResponse.php
index c58e6eaa..190c837b 100644
--- a/src/Chain/Toolbox/StreamResponse.php
+++ b/src/Chain/Toolbox/StreamResponse.php
@@ -5,14 +5,14 @@
 namespace PhpLlm\LlmChain\Chain\Toolbox;
 
 use PhpLlm\LlmChain\Model\Message\Message;
-use PhpLlm\LlmChain\Model\Response\ResponseInterface;
+use PhpLlm\LlmChain\Model\Response\BaseResponse;
 use PhpLlm\LlmChain\Model\Response\ToolCallResponse;
 
-final readonly class StreamResponse implements ResponseInterface
+final class StreamResponse extends BaseResponse
 {
     public function __construct(
-        private \Generator $generator,
-        private \Closure $handleToolCallsCallback,
+        private readonly \Generator $generator,
+        private readonly \Closure $handleToolCallsCallback,
     ) {
     }
 
diff --git a/src/Model/Response/AsyncResponse.php b/src/Model/Response/AsyncResponse.php
index b070a1cb..6eff7fb8 100644
--- a/src/Model/Response/AsyncResponse.php
+++ b/src/Model/Response/AsyncResponse.php
@@ -4,11 +4,15 @@
 
 namespace PhpLlm\LlmChain\Model\Response;
 
+use PhpLlm\LlmChain\Model\Response\Exception\RawResponseAlreadySet;
+use PhpLlm\LlmChain\Model\Response\Metadata\MetadataAwareTrait;
 use PhpLlm\LlmChain\Platform\ResponseConverter;
 use Symfony\Contracts\HttpClient\ResponseInterface as HttpResponse;
 
 final class AsyncResponse implements ResponseInterface
 {
+    use MetadataAwareTrait;
+
     private bool $isConverted = false;
     private ResponseInterface $convertedResponse;
 
@@ -27,10 +31,27 @@ public function getContent(): string|iterable|object|null
         return $this->unwrap()->getContent();
     }
 
+    public function getRawResponse(): HttpResponse
+    {
+        return $this->response;
+    }
+
+    public function setRawResponse(HttpResponse $rawResponse): void
+    {
+        // Empty by design as the raw response is already set in the constructor and must only be set once
+        throw new RawResponseAlreadySet();
+    }
+
     public function unwrap(): ResponseInterface
     {
         if (!$this->isConverted) {
             $this->convertedResponse = $this->responseConverter->convert($this->response, $this->options);
+
+            if (null === $this->convertedResponse->getRawResponse()) {
+                // Fallback to set the raw response when it was not handled by the response converter itself
+                $this->convertedResponse->setRawResponse($this->response);
+            }
+
             $this->isConverted = true;
         }
 
diff --git a/src/Model/Response/BaseResponse.php b/src/Model/Response/BaseResponse.php
new file mode 100644
index 00000000..b8848d2f
--- /dev/null
+++ b/src/Model/Response/BaseResponse.php
@@ -0,0 +1,13 @@
+<?php
+
+declare(strict_types=1);
+
+namespace PhpLlm\LlmChain\Model\Response;
+
+use PhpLlm\LlmChain\Model\Response\Metadata\MetadataAwareTrait;
+
+abstract class BaseResponse implements ResponseInterface
+{
+    use MetadataAwareTrait;
+    use RawResponseAwareTrait;
+}
diff --git a/src/Model/Response/BinaryResponse.php b/src/Model/Response/BinaryResponse.php
index b967ca47..2c0c4115 100644
--- a/src/Model/Response/BinaryResponse.php
+++ b/src/Model/Response/BinaryResponse.php
@@ -4,7 +4,7 @@
 
 namespace PhpLlm\LlmChain\Model\Response;
 
-final class BinaryResponse implements ResponseInterface
+final class BinaryResponse extends BaseResponse
 {
     public function __construct(
         public string $data,
diff --git a/src/Model/Response/ChoiceResponse.php b/src/Model/Response/ChoiceResponse.php
index 3f0a6d1b..760cab9f 100644
--- a/src/Model/Response/ChoiceResponse.php
+++ b/src/Model/Response/ChoiceResponse.php
@@ -6,12 +6,12 @@
 
 use PhpLlm\LlmChain\Exception\InvalidArgumentException;
 
-final readonly class ChoiceResponse implements ResponseInterface
+final class ChoiceResponse extends BaseResponse
 {
     /**
      * @var Choice[]
      */
-    private array $choices;
+    private readonly array $choices;
 
     public function __construct(Choice ...$choices)
     {
diff --git a/src/Model/Response/Exception/RawResponseAlreadySet.php b/src/Model/Response/Exception/RawResponseAlreadySet.php
new file mode 100644
index 00000000..6c9a436c
--- /dev/null
+++ b/src/Model/Response/Exception/RawResponseAlreadySet.php
@@ -0,0 +1,13 @@
+<?php
+
+declare(strict_types=1);
+
+namespace PhpLlm\LlmChain\Model\Response\Exception;
+
+final class RawResponseAlreadySet extends \RuntimeException
+{
+    public function __construct()
+    {
+        parent::__construct('The raw response was already set.');
+    }
+}
diff --git a/src/Model/Response/Metadata/Metadata.php b/src/Model/Response/Metadata/Metadata.php
new file mode 100644
index 00000000..9ec17fe3
--- /dev/null
+++ b/src/Model/Response/Metadata/Metadata.php
@@ -0,0 +1,99 @@
+<?php
+
+declare(strict_types=1);
+
+namespace PhpLlm\LlmChain\Model\Response\Metadata;
+
+/**
+ * @implements \IteratorAggregate<string, mixed>
+ * @implements \ArrayAccess<string, mixed>
+ */
+class Metadata implements \JsonSerializable, \Countable, \IteratorAggregate, \ArrayAccess
+{
+    /**
+     * @var array<string, mixed>
+     */
+    private array $metadata = [];
+
+    /**
+     * @param array<string, mixed> $metadata
+     */
+    public function __construct(array $metadata = [])
+    {
+        $this->set($metadata);
+    }
+
+    /**
+     * @return array<string, mixed>
+     */
+    public function all(): array
+    {
+        return $this->metadata;
+    }
+
+    /**
+     * @param array<string, mixed> $metadata
+     */
+    public function set(array $metadata): void
+    {
+        $this->metadata = $metadata;
+    }
+
+    public function add(string $key, mixed $value): void
+    {
+        $this->metadata[$key] = $value;
+    }
+
+    public function has(string $key): bool
+    {
+        return \array_key_exists($key, $this->metadata);
+    }
+
+    public function get(string $key, mixed $default = null): mixed
+    {
+        return $this->metadata[$key] ?? $default;
+    }
+
+    public function remove(string $key): void
+    {
+        unset($this->metadata[$key]);
+    }
+
+    /**
+     * @return array<string, mixed>
+     */
+    public function jsonSerialize(): array
+    {
+        return $this->all();
+    }
+
+    public function count(): int
+    {
+        return \count($this->metadata);
+    }
+
+    public function getIterator(): \Traversable
+    {
+        return new \ArrayIterator($this->metadata);
+    }
+
+    public function offsetExists(mixed $offset): bool
+    {
+        return $this->has((string) $offset);
+    }
+
+    public function offsetGet(mixed $offset): mixed
+    {
+        return $this->get((string) $offset);
+    }
+
+    public function offsetSet(mixed $offset, mixed $value): void
+    {
+        $this->add((string) $offset, $value);
+    }
+
+    public function offsetUnset(mixed $offset): void
+    {
+        $this->remove((string) $offset);
+    }
+}
diff --git a/src/Model/Response/Metadata/MetadataAwareTrait.php b/src/Model/Response/Metadata/MetadataAwareTrait.php
new file mode 100644
index 00000000..eac0f88c
--- /dev/null
+++ b/src/Model/Response/Metadata/MetadataAwareTrait.php
@@ -0,0 +1,15 @@
+<?php
+
+declare(strict_types=1);
+
+namespace PhpLlm\LlmChain\Model\Response\Metadata;
+
+trait MetadataAwareTrait
+{
+    private ?Metadata $metadata = null;
+
+    public function getMetadata(): Metadata
+    {
+        return $this->metadata ??= new Metadata();
+    }
+}
diff --git a/src/Model/Response/RawResponseAwareTrait.php b/src/Model/Response/RawResponseAwareTrait.php
new file mode 100644
index 00000000..29578dd3
--- /dev/null
+++ b/src/Model/Response/RawResponseAwareTrait.php
@@ -0,0 +1,27 @@
+<?php
+
+declare(strict_types=1);
+
+namespace PhpLlm\LlmChain\Model\Response;
+
+use PhpLlm\LlmChain\Model\Response\Exception\RawResponseAlreadySet;
+use Symfony\Contracts\HttpClient\ResponseInterface as SymfonyHttpResponse;
+
+trait RawResponseAwareTrait
+{
+    protected ?SymfonyHttpResponse $rawResponse = null;
+
+    public function setRawResponse(SymfonyHttpResponse $rawResponse): void
+    {
+        if (null !== $this->rawResponse) {
+            throw new RawResponseAlreadySet();
+        }
+
+        $this->rawResponse = $rawResponse;
+    }
+
+    public function getRawResponse(): ?SymfonyHttpResponse
+    {
+        return $this->rawResponse;
+    }
+}
diff --git a/src/Model/Response/ResponseInterface.php b/src/Model/Response/ResponseInterface.php
index 8b799bc6..a1651b79 100644
--- a/src/Model/Response/ResponseInterface.php
+++ b/src/Model/Response/ResponseInterface.php
@@ -4,10 +4,23 @@
 
 namespace PhpLlm\LlmChain\Model\Response;
 
+use PhpLlm\LlmChain\Model\Response\Exception\RawResponseAlreadySet;
+use PhpLlm\LlmChain\Model\Response\Metadata\Metadata;
+use Symfony\Contracts\HttpClient\ResponseInterface as SymfonyHttpResponse;
+
 interface ResponseInterface
 {
     /**
      * @return string|iterable<mixed>|object|null
      */
     public function getContent(): string|iterable|object|null;
+
+    public function getMetadata(): Metadata;
+
+    public function getRawResponse(): ?SymfonyHttpResponse;
+
+    /**
+     * @throws RawResponseAlreadySet if the response is tried to be set more than once
+     */
+    public function setRawResponse(SymfonyHttpResponse $rawResponse): void;
 }
diff --git a/src/Model/Response/StreamResponse.php b/src/Model/Response/StreamResponse.php
index b9618f54..da45ef23 100644
--- a/src/Model/Response/StreamResponse.php
+++ b/src/Model/Response/StreamResponse.php
@@ -4,10 +4,10 @@
 
 namespace PhpLlm\LlmChain\Model\Response;
 
-final readonly class StreamResponse implements ResponseInterface
+final class StreamResponse extends BaseResponse
 {
     public function __construct(
-        private \Generator $generator,
+        private readonly \Generator $generator,
     ) {
     }
 
diff --git a/src/Model/Response/StructuredResponse.php b/src/Model/Response/StructuredResponse.php
index 25918ca7..96c53123 100644
--- a/src/Model/Response/StructuredResponse.php
+++ b/src/Model/Response/StructuredResponse.php
@@ -4,13 +4,13 @@
 
 namespace PhpLlm\LlmChain\Model\Response;
 
-final readonly class StructuredResponse implements ResponseInterface
+final class StructuredResponse extends BaseResponse
 {
     /**
      * @param object|array<string, mixed> $structuredOutput
      */
     public function __construct(
-        private object|array $structuredOutput,
+        private readonly object|array $structuredOutput,
     ) {
     }
 
diff --git a/src/Model/Response/TextResponse.php b/src/Model/Response/TextResponse.php
index 1c289e31..09c37de7 100644
--- a/src/Model/Response/TextResponse.php
+++ b/src/Model/Response/TextResponse.php
@@ -4,10 +4,10 @@
 
 namespace PhpLlm\LlmChain\Model\Response;
 
-final readonly class TextResponse implements ResponseInterface
+final class TextResponse extends BaseResponse
 {
     public function __construct(
-        private string $content,
+        private readonly string $content,
     ) {
     }
 
diff --git a/src/Model/Response/ToolCallResponse.php b/src/Model/Response/ToolCallResponse.php
index 40bff942..0f4530d0 100644
--- a/src/Model/Response/ToolCallResponse.php
+++ b/src/Model/Response/ToolCallResponse.php
@@ -6,12 +6,12 @@
 
 use PhpLlm\LlmChain\Exception\InvalidArgumentException;
 
-final readonly class ToolCallResponse implements ResponseInterface
+final class ToolCallResponse extends BaseResponse
 {
     /**
      * @var ToolCall[]
      */
-    private array $toolCalls;
+    private readonly array $toolCalls;
 
     public function __construct(ToolCall ...$toolCalls)
     {
diff --git a/src/Model/Response/VectorResponse.php b/src/Model/Response/VectorResponse.php
index f14201eb..9e54a704 100644
--- a/src/Model/Response/VectorResponse.php
+++ b/src/Model/Response/VectorResponse.php
@@ -6,12 +6,12 @@
 
 use PhpLlm\LlmChain\Document\Vector;
 
-final readonly class VectorResponse implements ResponseInterface
+final class VectorResponse extends BaseResponse
 {
     /**
      * @var Vector[]
      */
-    private array $vectors;
+    private readonly array $vectors;
 
     public function __construct(Vector ...$vector)
     {
diff --git a/tests/Bridge/OpenAI/TokenOutputProcessorTest.php b/tests/Bridge/OpenAI/TokenOutputProcessorTest.php
new file mode 100644
index 00000000..eb4d864e
--- /dev/null
+++ b/tests/Bridge/OpenAI/TokenOutputProcessorTest.php
@@ -0,0 +1,152 @@
+<?php
+
+declare(strict_types=1);
+
+namespace PhpLlm\LlmChain\Tests\Bridge\OpenAI;
+
+use PhpLlm\LlmChain\Bridge\OpenAI\TokenOutputProcessor;
+use PhpLlm\LlmChain\Chain\Output;
+use PhpLlm\LlmChain\Model\LanguageModel;
+use PhpLlm\LlmChain\Model\Message\MessageBagInterface;
+use PhpLlm\LlmChain\Model\Response\Metadata\Metadata;
+use PhpLlm\LlmChain\Model\Response\ResponseInterface;
+use PhpLlm\LlmChain\Model\Response\StreamResponse;
+use PhpLlm\LlmChain\Model\Response\TextResponse;
+use PHPUnit\Framework\Attributes\CoversClass;
+use PHPUnit\Framework\Attributes\Small;
+use PHPUnit\Framework\Attributes\Test;
+use PHPUnit\Framework\Attributes\UsesClass;
+use PHPUnit\Framework\TestCase;
+use Symfony\Contracts\HttpClient\ResponseInterface as SymfonyHttpResponse;
+
+#[CoversClass(TokenOutputProcessor::class)]
+#[UsesClass(Output::class)]
+#[UsesClass(TextResponse::class)]
+#[UsesClass(StreamResponse::class)]
+#[UsesClass(Metadata::class)]
+#[UsesClass(SymfonyHttpResponse::class)]
+#[Small]
+final class TokenOutputProcessorTest extends TestCase
+{
+    #[Test]
+    public function itHandlesStreamResponsesWithoutProcessing(): void
+    {
+        $processor = new TokenOutputProcessor();
+        $streamResponse = new StreamResponse((static function () { yield 'test'; })());
+        $output = $this->createOutput($streamResponse);
+
+        $processor->processOutput($output);
+
+        $metadata = $output->response->getMetadata();
+        self::assertCount(0, $metadata);
+    }
+
+    #[Test]
+    public function itDoesNothingWithoutRawResponse(): void
+    {
+        $processor = new TokenOutputProcessor();
+        $textResponse = new TextResponse('test');
+        $output = $this->createOutput($textResponse);
+
+        $processor->processOutput($output);
+
+        $metadata = $output->response->getMetadata();
+        self::assertCount(0, $metadata);
+    }
+
+    #[Test]
+    public function itAddsRemainingTokensToMetadata(): void
+    {
+        $processor = new TokenOutputProcessor();
+        $textResponse = new TextResponse('test');
+
+        $rawResponse = self::createStub(SymfonyHttpResponse::class);
+        $rawResponse->method('getHeaders')->willReturn([
+            'x-ratelimit-remaining-tokens' => ['1000'],
+        ]);
+        $rawResponse->method('toArray')->willReturn([]);
+
+        $textResponse->setRawResponse($rawResponse);
+
+        $output = $this->createOutput($textResponse);
+
+        $processor->processOutput($output);
+
+        $metadata = $output->response->getMetadata();
+        self::assertCount(1, $metadata);
+        self::assertSame(1000, $metadata->get('remaining_tokens'));
+    }
+
+    #[Test]
+    public function itAddsUsageTokensToMetadata(): void
+    {
+        $processor = new TokenOutputProcessor();
+        $textResponse = new TextResponse('test');
+
+        $rawResponse = self::createStub(SymfonyHttpResponse::class);
+        $rawResponse->method('getHeaders')->willReturn([
+            'x-ratelimit-remaining-tokens' => ['1000'],
+        ]);
+        $rawResponse->method('toArray')->willReturn([
+            'usage' => [
+                'prompt_tokens' => 10,
+                'completion_tokens' => 20,
+                'total_tokens' => 30,
+            ],
+        ]);
+
+        $textResponse->setRawResponse($rawResponse);
+
+        $output = $this->createOutput($textResponse);
+
+        $processor->processOutput($output);
+
+        $metadata = $output->response->getMetadata();
+        self::assertCount(4, $metadata);
+        self::assertSame(1000, $metadata->get('remaining_tokens'));
+        self::assertSame(10, $metadata->get('prompt_tokens'));
+        self::assertSame(20, $metadata->get('completion_tokens'));
+        self::assertSame(30, $metadata->get('total_tokens'));
+    }
+
+    #[Test]
+    public function itHandlesMissingUsageFields(): void
+    {
+        $processor = new TokenOutputProcessor();
+        $textResponse = new TextResponse('test');
+
+        $rawResponse = self::createStub(SymfonyHttpResponse::class);
+        $rawResponse->method('getHeaders')->willReturn([
+            'x-ratelimit-remaining-tokens' => ['1000'],
+        ]);
+        $rawResponse->method('toArray')->willReturn([
+            'usage' => [
+                // Missing some fields
+                'prompt_tokens' => 10,
+            ],
+        ]);
+
+        $textResponse->setRawResponse($rawResponse);
+
+        $output = $this->createOutput($textResponse);
+
+        $processor->processOutput($output);
+
+        $metadata = $output->response->getMetadata();
+        self::assertCount(4, $metadata);
+        self::assertSame(1000, $metadata->get('remaining_tokens'));
+        self::assertSame(10, $metadata->get('prompt_tokens'));
+        self::assertNull($metadata->get('completion_tokens'));
+        self::assertNull($metadata->get('total_tokens'));
+    }
+
+    private function createOutput(ResponseInterface $response): Output
+    {
+        return new Output(
+            self::createStub(LanguageModel::class),
+            $response,
+            self::createStub(MessageBagInterface::class),
+            [],
+        );
+    }
+}
diff --git a/tests/Model/Response/AsyncResponseTest.php b/tests/Model/Response/AsyncResponseTest.php
new file mode 100644
index 00000000..71143382
--- /dev/null
+++ b/tests/Model/Response/AsyncResponseTest.php
@@ -0,0 +1,162 @@
+<?php
+
+declare(strict_types=1);
+
+namespace PhpLlm\LlmChain\Tests\Model\Response;
+
+use PhpLlm\LlmChain\Model\Response\AsyncResponse;
+use PhpLlm\LlmChain\Model\Response\BaseResponse;
+use PhpLlm\LlmChain\Model\Response\Exception\RawResponseAlreadySet;
+use PhpLlm\LlmChain\Model\Response\Metadata\Metadata;
+use PhpLlm\LlmChain\Model\Response\ResponseInterface;
+use PhpLlm\LlmChain\Model\Response\TextResponse;
+use PhpLlm\LlmChain\Platform\ResponseConverter;
+use PHPUnit\Framework\Attributes\CoversClass;
+use PHPUnit\Framework\Attributes\Small;
+use PHPUnit\Framework\Attributes\Test;
+use PHPUnit\Framework\Attributes\UsesClass;
+use PHPUnit\Framework\TestCase;
+use Symfony\Contracts\HttpClient\ResponseInterface as SymfonyHttpResponse;
+
+#[CoversClass(AsyncResponse::class)]
+#[UsesClass(Metadata::class)]
+#[UsesClass(TextResponse::class)]
+#[UsesClass(RawResponseAlreadySet::class)]
+#[Small]
+final class AsyncResponseTest extends TestCase
+{
+    #[Test]
+    public function itUnwrapsTheResponseWhenGettingContent(): void
+    {
+        $httpResponse = $this->createStub(SymfonyHttpResponse::class);
+        $textResponse = new TextResponse('test content');
+
+        $responseConverter = $this->createMock(ResponseConverter::class);
+        $responseConverter->expects($this->once())
+            ->method('convert')
+            ->with($httpResponse, [])
+            ->willReturn($textResponse);
+
+        $asyncResponse = new AsyncResponse($responseConverter, $httpResponse);
+
+        self::assertSame('test content', $asyncResponse->getContent());
+    }
+
+    #[Test]
+    public function itConvertsTheResponseOnlyOnce(): void
+    {
+        $httpResponse = $this->createStub(SymfonyHttpResponse::class);
+        $textResponse = new TextResponse('test content');
+
+        $responseConverter = $this->createMock(ResponseConverter::class);
+        $responseConverter->expects($this->once())
+            ->method('convert')
+            ->with($httpResponse, [])
+            ->willReturn($textResponse);
+
+        $asyncResponse = new AsyncResponse($responseConverter, $httpResponse);
+
+        // Call unwrap multiple times, but the converter should only be called once
+        $asyncResponse->unwrap();
+        $asyncResponse->unwrap();
+        $asyncResponse->getContent();
+    }
+
+    #[Test]
+    public function itGetsRawResponseDirectly(): void
+    {
+        $httpResponse = $this->createStub(SymfonyHttpResponse::class);
+        $responseConverter = $this->createStub(ResponseConverter::class);
+
+        $asyncResponse = new AsyncResponse($responseConverter, $httpResponse);
+
+        self::assertSame($httpResponse, $asyncResponse->getRawResponse());
+    }
+
+    #[Test]
+    public function itThrowsExceptionWhenSettingRawResponse(): void
+    {
+        $this->expectException(RawResponseAlreadySet::class);
+
+        $httpResponse = $this->createStub(SymfonyHttpResponse::class);
+        $responseConverter = $this->createStub(ResponseConverter::class);
+
+        $asyncResponse = new AsyncResponse($responseConverter, $httpResponse);
+        $asyncResponse->setRawResponse($httpResponse);
+    }
+
+    #[Test]
+    public function itSetsRawResponseOnUnwrappedResponseWhenNeeded(): void
+    {
+        $httpResponse = $this->createStub(SymfonyHttpResponse::class);
+
+        $unwrappedResponse = $this->createResponse(null);
+
+        $responseConverter = $this->createStub(ResponseConverter::class);
+        $responseConverter->method('convert')->willReturn($unwrappedResponse);
+
+        $asyncResponse = new AsyncResponse($responseConverter, $httpResponse);
+        $asyncResponse->unwrap();
+
+        // The raw response in the model response is now set and not null anymore
+        self::assertSame($httpResponse, $unwrappedResponse->getRawResponse());
+    }
+
+    #[Test]
+    public function itDoesNotSetRawResponseOnUnwrappedResponseWhenAlreadySet(): void
+    {
+        $originHttpResponse = $this->createStub(SymfonyHttpResponse::class);
+        $anotherHttpResponse = $this->createStub(SymfonyHttpResponse::class);
+
+        $unwrappedResponse = $this->createResponse($anotherHttpResponse);
+
+        $responseConverter = $this->createStub(ResponseConverter::class);
+        $responseConverter->method('convert')->willReturn($unwrappedResponse);
+
+        $asyncResponse = new AsyncResponse($responseConverter, $originHttpResponse);
+        $asyncResponse->unwrap();
+
+        // It is still the same raw response as set initially and so not overwritten
+        self::assertSame($anotherHttpResponse, $unwrappedResponse->getRawResponse());
+    }
+
+    /**
+     * Workaround for low deps because mocking the ResponseInterface leads to an exception with
+     * mock creation "Type Traversable|object|array|string|null contains both object and a class type"
+     * in PHPUnit MockClass.
+     */
+    private function createResponse(?SymfonyHttpResponse $rawResponse): ResponseInterface
+    {
+        return new class($rawResponse) extends BaseResponse {
+            public function __construct(protected ?SymfonyHttpResponse $rawResponse)
+            {
+            }
+
+            public function getContent(): string
+            {
+                return 'test content';
+            }
+
+            public function getRawResponse(): ?SymfonyHttpResponse
+            {
+                return $this->rawResponse;
+            }
+        };
+    }
+
+    #[Test]
+    public function itPassesOptionsToConverter(): void
+    {
+        $httpResponse = $this->createStub(SymfonyHttpResponse::class);
+        $options = ['option1' => 'value1', 'option2' => 'value2'];
+
+        $responseConverter = $this->createMock(ResponseConverter::class);
+        $responseConverter->expects($this->once())
+            ->method('convert')
+            ->with($httpResponse, $options)
+            ->willReturn($this->createResponse(null));
+
+        $asyncResponse = new AsyncResponse($responseConverter, $httpResponse, $options);
+        $asyncResponse->unwrap();
+    }
+}
diff --git a/tests/Model/Response/BaseResponseTest.php b/tests/Model/Response/BaseResponseTest.php
new file mode 100644
index 00000000..2988d694
--- /dev/null
+++ b/tests/Model/Response/BaseResponseTest.php
@@ -0,0 +1,69 @@
+<?php
+
+declare(strict_types=1);
+
+namespace PhpLlm\LlmChain\Tests\Model\Response;
+
+use PhpLlm\LlmChain\Model\Response\BaseResponse;
+use PhpLlm\LlmChain\Model\Response\Exception\RawResponseAlreadySet;
+use PhpLlm\LlmChain\Model\Response\Metadata\MetadataAwareTrait;
+use PhpLlm\LlmChain\Model\Response\RawResponseAwareTrait;
+use PHPUnit\Framework\Attributes\CoversClass;
+use PHPUnit\Framework\Attributes\Small;
+use PHPUnit\Framework\Attributes\Test;
+use PHPUnit\Framework\Attributes\UsesTrait;
+use PHPUnit\Framework\TestCase;
+use Symfony\Contracts\HttpClient\ResponseInterface as SymfonyHttpResponse;
+
+#[CoversClass(BaseResponse::class)]
+#[UsesTrait(MetadataAwareTrait::class)]
+#[UsesTrait(RawResponseAwareTrait::class)]
+#[Small]
+final class BaseResponseTest extends TestCase
+{
+    #[Test]
+    public function itCanHandleMetadata(): void
+    {
+        $response = $this->createResponse();
+        $metadata = $response->getMetadata();
+
+        self::assertCount(0, $metadata);
+
+        $metadata->add('key', 'value');
+        $metadata = $response->getMetadata();
+
+        self::assertCount(1, $metadata);
+    }
+
+    #[Test]
+    public function itCanBeEnrichedWithARawResponse(): void
+    {
+        $response = $this->createResponse();
+        $rawResponse = $this->createMock(SymfonyHttpResponse::class);
+
+        $response->setRawResponse($rawResponse);
+        self::assertSame($rawResponse, $response->getRawResponse());
+    }
+
+    #[Test]
+    public function itThrowsAnExceptionWhenSettingARawResponseTwice(): void
+    {
+        $this->expectException(RawResponseAlreadySet::class);
+
+        $response = $this->createResponse();
+        $rawResponse = $this->createMock(SymfonyHttpResponse::class);
+
+        $response->setRawResponse($rawResponse);
+        $response->setRawResponse($rawResponse);
+    }
+
+    private function createResponse(): BaseResponse
+    {
+        return new class extends BaseResponse {
+            public function getContent(): string
+            {
+                return 'test';
+            }
+        };
+    }
+}
diff --git a/tests/Model/Response/Exception/RawResponseAlreadySetTest.php b/tests/Model/Response/Exception/RawResponseAlreadySetTest.php
new file mode 100644
index 00000000..42d35744
--- /dev/null
+++ b/tests/Model/Response/Exception/RawResponseAlreadySetTest.php
@@ -0,0 +1,24 @@
+<?php
+
+declare(strict_types=1);
+
+namespace PhpLlm\LlmChain\Tests\Model\Response\Exception;
+
+use PhpLlm\LlmChain\Model\Response\Exception\RawResponseAlreadySet;
+use PHPUnit\Framework\Attributes\CoversClass;
+use PHPUnit\Framework\Attributes\Small;
+use PHPUnit\Framework\Attributes\Test;
+use PHPUnit\Framework\TestCase;
+
+#[CoversClass(RawResponseAlreadySet::class)]
+#[Small]
+final class RawResponseAlreadySetTest extends TestCase
+{
+    #[Test]
+    public function itHasCorrectExceptionMessage(): void
+    {
+        $exception = new RawResponseAlreadySet();
+
+        self::assertSame('The raw response was already set.', $exception->getMessage());
+    }
+}
diff --git a/tests/Model/Response/Metadata/MetadataAwareTraitTest.php b/tests/Model/Response/Metadata/MetadataAwareTraitTest.php
new file mode 100644
index 00000000..c1f5543f
--- /dev/null
+++ b/tests/Model/Response/Metadata/MetadataAwareTraitTest.php
@@ -0,0 +1,37 @@
+<?php
+
+declare(strict_types=1);
+
+namespace PhpLlm\LlmChain\Tests\Model\Response\Metadata;
+
+use PhpLlm\LlmChain\Model\Response\Metadata\MetadataAwareTrait;
+use PHPUnit\Framework\Attributes\CoversTrait;
+use PHPUnit\Framework\Attributes\Small;
+use PHPUnit\Framework\Attributes\Test;
+use PHPUnit\Framework\TestCase;
+
+#[CoversTrait(MetadataAwareTrait::class)]
+#[Small]
+final class MetadataAwareTraitTest extends TestCase
+{
+    #[Test]
+    public function itCanHandleMetadata(): void
+    {
+        $response = $this->createTestClass();
+        $metadata = $response->getMetadata();
+
+        self::assertCount(0, $metadata);
+
+        $metadata->add('key', 'value');
+        $metadata = $response->getMetadata();
+
+        self::assertCount(1, $metadata);
+    }
+
+    private function createTestClass(): object
+    {
+        return new class {
+            use MetadataAwareTrait;
+        };
+    }
+}
diff --git a/tests/Model/Response/Metadata/MetadataTest.php b/tests/Model/Response/Metadata/MetadataTest.php
new file mode 100644
index 00000000..70081482
--- /dev/null
+++ b/tests/Model/Response/Metadata/MetadataTest.php
@@ -0,0 +1,130 @@
+<?php
+
+declare(strict_types=1);
+
+namespace PhpLlm\LlmChain\Tests\Model\Response\Metadata;
+
+use PhpLlm\LlmChain\Model\Response\Metadata\Metadata;
+use PHPUnit\Framework\Attributes\CoversClass;
+use PHPUnit\Framework\Attributes\Small;
+use PHPUnit\Framework\Attributes\Test;
+use PHPUnit\Framework\TestCase;
+
+#[CoversClass(Metadata::class)]
+#[Small]
+final class MetadataTest extends TestCase
+{
+    #[Test]
+    public function itCanBeCreatedEmpty(): void
+    {
+        $metadata = new Metadata();
+        self::assertCount(0, $metadata);
+        self::assertSame([], $metadata->all());
+    }
+
+    #[Test]
+    public function itCanBeCreatedWithInitialData(): void
+    {
+        $metadata = new Metadata(['key' => 'value']);
+        self::assertCount(1, $metadata);
+        self::assertSame(['key' => 'value'], $metadata->all());
+    }
+
+    #[Test]
+    public function itCanAddNewMetadata(): void
+    {
+        $metadata = new Metadata();
+        $metadata->add('key', 'value');
+
+        self::assertTrue($metadata->has('key'));
+        self::assertSame('value', $metadata->get('key'));
+    }
+
+    #[Test]
+    public function itCanCheckIfMetadataExists(): void
+    {
+        $metadata = new Metadata(['key' => 'value']);
+
+        self::assertTrue($metadata->has('key'));
+        self::assertFalse($metadata->has('nonexistent'));
+    }
+
+    #[Test]
+    public function itCanGetMetadataWithDefault(): void
+    {
+        $metadata = new Metadata(['key' => 'value']);
+
+        self::assertSame('value', $metadata->get('key'));
+        self::assertSame('default', $metadata->get('nonexistent', 'default'));
+        self::assertNull($metadata->get('nonexistent'));
+    }
+
+    #[Test]
+    public function itCanRemoveMetadata(): void
+    {
+        $metadata = new Metadata(['key' => 'value']);
+        self::assertTrue($metadata->has('key'));
+
+        $metadata->remove('key');
+        self::assertFalse($metadata->has('key'));
+    }
+
+    #[Test]
+    public function itCanSetEntireMetadataArray(): void
+    {
+        $metadata = new Metadata(['key1' => 'value1']);
+        $metadata->set(['key2' => 'value2', 'key3' => 'value3']);
+
+        self::assertFalse($metadata->has('key1'));
+        self::assertTrue($metadata->has('key2'));
+        self::assertTrue($metadata->has('key3'));
+        self::assertSame(['key2' => 'value2', 'key3' => 'value3'], $metadata->all());
+    }
+
+    #[Test]
+    public function itImplementsJsonSerializable(): void
+    {
+        $metadata = new Metadata(['key' => 'value']);
+        self::assertSame(['key' => 'value'], $metadata->jsonSerialize());
+    }
+
+    #[Test]
+    public function itImplementsArrayAccess(): void
+    {
+        $metadata = new Metadata(['key' => 'value']);
+
+        self::assertArrayHasKey('key', $metadata);
+        self::assertSame('value', $metadata['key']);
+
+        $metadata['new'] = 'newValue';
+        self::assertSame('newValue', $metadata['new']);
+
+        unset($metadata['key']);
+        self::assertArrayNotHasKey('key', $metadata);
+    }
+
+    #[Test]
+    public function itImplementsIteratorAggregate(): void
+    {
+        $metadata = new Metadata(['key1' => 'value1', 'key2' => 'value2']);
+        $result = \iterator_to_array($metadata);
+
+        self::assertSame(['key1' => 'value1', 'key2' => 'value2'], $result);
+    }
+
+    #[Test]
+    public function itImplementsCountable(): void
+    {
+        $metadata = new Metadata();
+        self::assertCount(0, $metadata);
+
+        $metadata->add('key', 'value');
+        self::assertCount(1, $metadata);
+
+        $metadata->add('key2', 'value2');
+        self::assertCount(2, $metadata);
+
+        $metadata->remove('key');
+        self::assertCount(1, $metadata);
+    }
+}
diff --git a/tests/Model/Response/RawResponseAwareTraitTest.php b/tests/Model/Response/RawResponseAwareTraitTest.php
new file mode 100644
index 00000000..1ad29208
--- /dev/null
+++ b/tests/Model/Response/RawResponseAwareTraitTest.php
@@ -0,0 +1,47 @@
+<?php
+
+declare(strict_types=1);
+
+namespace PhpLlm\LlmChain\Tests\Model\Response;
+
+use PhpLlm\LlmChain\Model\Response\Exception\RawResponseAlreadySet;
+use PhpLlm\LlmChain\Model\Response\RawResponseAwareTrait;
+use PHPUnit\Framework\Attributes\CoversTrait;
+use PHPUnit\Framework\Attributes\Small;
+use PHPUnit\Framework\Attributes\Test;
+use PHPUnit\Framework\TestCase;
+use Symfony\Contracts\HttpClient\ResponseInterface as SymfonyHttpResponse;
+
+#[CoversTrait(RawResponseAwareTrait::class)]
+#[Small]
+final class RawResponseAwareTraitTest extends TestCase
+{
+    #[Test]
+    public function itCanBeEnrichedWithARawResponse(): void
+    {
+        $response = $this->createTestClass();
+        $rawResponse = $this->createMock(SymfonyHttpResponse::class);
+
+        $response->setRawResponse($rawResponse);
+        self::assertSame($rawResponse, $response->getRawResponse());
+    }
+
+    #[Test]
+    public function itThrowsAnExceptionWhenSettingARawResponseTwice(): void
+    {
+        $this->expectException(RawResponseAlreadySet::class);
+
+        $response = $this->createTestClass();
+        $rawResponse = $this->createMock(SymfonyHttpResponse::class);
+
+        $response->setRawResponse($rawResponse);
+        $response->setRawResponse($rawResponse);
+    }
+
+    private function createTestClass(): object
+    {
+        return new class {
+            use RawResponseAwareTrait;
+        };
+    }
+}


Reviews:
user: login: OskarStark
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T12:51:20Z
user: login: chr-hertel
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T19:17:13Z
user: login: chr-hertel
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T19:22:27Z
user: login: chr-hertel
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T19:25:16Z
user: login: chr-hertel
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-22T19:25:56Z
user: login: DZunke
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T05:21:58Z
user: login: DZunke
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T05:24:44Z
user: login: DZunke
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T05:25:18Z
user: login: OskarStark
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-24T09:05:50Z
user: login: chr-hertel
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-25T08:20:22Z
user: login: chr-hertel
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-25T08:28:28Z
user: login: chr-hertel
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-25T08:32:02Z
user: login: chr-hertel
body: Thanks a ton for taking care about this @DZunke - that feature was overdue!
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-25T08:38:27Z

Review comments:
diff_hunk: @@ -0,0 +1,37 @@
+<?php
+
+declare(strict_types=1);
+
+namespace PhpLlm\LlmChain\Bridge\OpenAI;
+
+use PhpLlm\LlmChain\Chain\Output;
+use PhpLlm\LlmChain\Chain\OutputProcessor;
+
+final class TokenOutputProcessor implements OutputProcessor
+{
+    public function processOutput(Output $output): void
+    {
+        $rawResponse = $output->response->getRawResponse();
+        if (null === $rawResponse) {
+            return;
+        }
+
+        $metadata = $output->response->getMetadata();
+
+        $metadata->add(
+            'remaining_tokens',
+            (int) $rawResponse->getHeaders(false)['x-ratelimit-remaining-tokens'][0],
+        );
+
+        $content = $rawResponse->getContent(false);
+        $content = \json_decode($content, true);
+
+        if (!\array_key_exists('usage', $content)) {
+            return;
+        }
+
+        $metadata->add('prompt_tokens', $content['usage']['prompt_tokens'] ?? null);
+        $metadata->add('completion_tokens', $content['usage']['completion_tokens'] ?? null);
+        $metadata->add('total_tokens', $content['usage']['total_tokens'] ?? null);
path: src/Bridge/OpenAI/TokenOutputProcessor.php
user: login: OskarStark
body: I think it could be handy to be able to provide an array

```php
$metadata->add([
    'prompt_tokens' => ......,
]);
```

created_at: 2025-04-21T12:51:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,19 @@
+<?php
+
+declare(strict_types=1);
+
+namespace PhpLlm\LlmChain\Model\Response\Metadata;
+
+trait EmptyMetadataTrait
path: src/Model/Response/Metadata/EmptyMetadataTrait.php
user: login: chr-hertel
body: to me it would be more consistent to be also `MetadataAwareTrait` - sounds a bit like metadata would be empty when using this one, but in reality it's just initializing that
created_at: 2025-04-22T19:17:13Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,43 @@
+<?php
+
+declare(strict_types=1);
+
+namespace PhpLlm\LlmChain\Bridge\OpenAI;
+
+use PhpLlm\LlmChain\Chain\Output;
+use PhpLlm\LlmChain\Chain\OutputProcessor;
+use PhpLlm\LlmChain\Model\Response\StreamResponse;
+
+final class TokenOutputProcessor implements OutputProcessor
+{
+    public function processOutput(Output $output): void
+    {
+        if ($output->response instanceof StreamResponse) {
+            // Streams have to be handled manually as the tokens are part of the streamed chunks
+            return;
+        }
+
+        $rawResponse = $output->response->getRawResponse();
+        if (null === $rawResponse) {
+            return;
+        }
+
+        $metadata = $output->response->getMetadata();
+
+        $metadata->add(
+            'remaining_tokens',
+            (int) $rawResponse->getHeaders(false)['x-ratelimit-remaining-tokens'][0],
+        );
+
+        $content = $rawResponse->getContent(false);
+        $content = \json_decode($content, true);
path: src/Bridge/OpenAI/TokenOutputProcessor.php
user: login: chr-hertel
body: ```suggestion
        $content = $rawResponse->toArray(false);
```
created_at: 2025-04-22T19:22:27Z
author_association: MEMBER
diff_hunk: @@ -27,10 +31,27 @@ public function getContent(): string|iterable|object|null
         return $this->unwrap()->getContent();
     }
 
+    public function getRawResponse(): HttpResponse
+    {
+        return $this->response;
+    }
+
+    public function setRawResponse(HttpResponse $rawResponse): void
+    {
+        // Empty by design as the raw response is already set in the constructor and must only be set once
+        throw new RawResponseAlreadySet();
+    }
+
     public function unwrap(): ResponseInterface
     {
         if (!$this->isConverted) {
             $this->convertedResponse = $this->responseConverter->convert($this->response, $this->options);
+
+            if (null === $this->convertedResponse->getRawResponse()) {
+                // Fallback to set the raw response when it was not handled by the response converter itself
+                $this->convertedResponse->setRawResponse($this->response);
path: src/Model/Response/AsyncResponse.php
user: login: chr-hertel
body: this is currently the path for the most cases, right?
created_at: 2025-04-22T19:25:16Z
author_association: MEMBER
path: src/Model/Response/Metadata/Metadata.php
user: login: chr-hertel
body: smooth :) :+1: 
created_at: 2025-04-22T19:25:55Z
author_association: MEMBER
diff_hunk: @@ -27,10 +31,27 @@ public function getContent(): string|iterable|object|null
         return $this->unwrap()->getContent();
     }
 
+    public function getRawResponse(): HttpResponse
+    {
+        return $this->response;
+    }
+
+    public function setRawResponse(HttpResponse $rawResponse): void
+    {
+        // Empty by design as the raw response is already set in the constructor and must only be set once
+        throw new RawResponseAlreadySet();
+    }
+
     public function unwrap(): ResponseInterface
     {
         if (!$this->isConverted) {
             $this->convertedResponse = $this->responseConverter->convert($this->response, $this->options);
+
+            if (null === $this->convertedResponse->getRawResponse()) {
+                // Fallback to set the raw response when it was not handled by the response converter itself
+                $this->convertedResponse->setRawResponse($this->response);
path: src/Model/Response/AsyncResponse.php
user: login: DZunke
body: Yeah. I just wanted to open up that people can do it also in the response converter for themself ... you never know. 
created_at: 2025-04-24T05:21:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,37 @@
+<?php
+
+declare(strict_types=1);
+
+namespace PhpLlm\LlmChain\Bridge\OpenAI;
+
+use PhpLlm\LlmChain\Chain\Output;
+use PhpLlm\LlmChain\Chain\OutputProcessor;
+
+final class TokenOutputProcessor implements OutputProcessor
+{
+    public function processOutput(Output $output): void
+    {
+        $rawResponse = $output->response->getRawResponse();
+        if (null === $rawResponse) {
+            return;
+        }
+
+        $metadata = $output->response->getMetadata();
+
+        $metadata->add(
+            'remaining_tokens',
+            (int) $rawResponse->getHeaders(false)['x-ratelimit-remaining-tokens'][0],
+        );
+
+        $content = $rawResponse->getContent(false);
+        $content = \json_decode($content, true);
+
+        if (!\array_key_exists('usage', $content)) {
+            return;
+        }
+
+        $metadata->add('prompt_tokens', $content['usage']['prompt_tokens'] ?? null);
+        $metadata->add('completion_tokens', $content['usage']['completion_tokens'] ?? null);
+        $metadata->add('total_tokens', $content['usage']['total_tokens'] ?? null);
path: src/Bridge/OpenAI/TokenOutputProcessor.php
user: login: DZunke
body: I think this is a complicated one. I would expect an `add` to a collection is just taking a single entry. Naming things 😁 I first thought about a `merge` but then i would expect another Metadata object as argument. Maybe something like `addArray` ? 
created_at: 2025-04-24T05:24:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,19 @@
+<?php
+
+declare(strict_types=1);
+
+namespace PhpLlm\LlmChain\Model\Response\Metadata;
+
+trait EmptyMetadataTrait
path: src/Model/Response/Metadata/EmptyMetadataTrait.php
user: login: DZunke
body: Thanks! You are right, i rename it 👍🏼 
created_at: 2025-04-24T05:25:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,37 @@
+<?php
+
+declare(strict_types=1);
+
+namespace PhpLlm\LlmChain\Bridge\OpenAI;
+
+use PhpLlm\LlmChain\Chain\Output;
+use PhpLlm\LlmChain\Chain\OutputProcessor;
+
+final class TokenOutputProcessor implements OutputProcessor
+{
+    public function processOutput(Output $output): void
+    {
+        $rawResponse = $output->response->getRawResponse();
+        if (null === $rawResponse) {
+            return;
+        }
+
+        $metadata = $output->response->getMetadata();
+
+        $metadata->add(
+            'remaining_tokens',
+            (int) $rawResponse->getHeaders(false)['x-ratelimit-remaining-tokens'][0],
+        );
+
+        $content = $rawResponse->getContent(false);
+        $content = \json_decode($content, true);
+
+        if (!\array_key_exists('usage', $content)) {
+            return;
+        }
+
+        $metadata->add('prompt_tokens', $content['usage']['prompt_tokens'] ?? null);
+        $metadata->add('completion_tokens', $content['usage']['completion_tokens'] ?? null);
+        $metadata->add('total_tokens', $content['usage']['total_tokens'] ?? null);
path: src/Bridge/OpenAI/TokenOutputProcessor.php
user: login: OskarStark
body: what about `function add(string|array $value, mixed $content)` while content should be ignored then ofc.
I don't like the fuzzy syntax, but have no other ideas, maybe let's skip the idea completely for now @chr-hertel ?
created_at: 2025-04-24T09:05:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,37 @@
+<?php
+
+declare(strict_types=1);
+
+namespace PhpLlm\LlmChain\Bridge\OpenAI;
+
+use PhpLlm\LlmChain\Chain\Output;
+use PhpLlm\LlmChain\Chain\OutputProcessor;
+
+final class TokenOutputProcessor implements OutputProcessor
+{
+    public function processOutput(Output $output): void
+    {
+        $rawResponse = $output->response->getRawResponse();
+        if (null === $rawResponse) {
+            return;
+        }
+
+        $metadata = $output->response->getMetadata();
+
+        $metadata->add(
+            'remaining_tokens',
+            (int) $rawResponse->getHeaders(false)['x-ratelimit-remaining-tokens'][0],
+        );
+
+        $content = $rawResponse->getContent(false);
+        $content = \json_decode($content, true);
+
+        if (!\array_key_exists('usage', $content)) {
+            return;
+        }
+
+        $metadata->add('prompt_tokens', $content['usage']['prompt_tokens'] ?? null);
+        $metadata->add('completion_tokens', $content['usage']['completion_tokens'] ?? null);
+        $metadata->add('total_tokens', $content['usage']['total_tokens'] ?? null);
path: src/Bridge/OpenAI/TokenOutputProcessor.php
user: login: chr-hertel
body: really two sided here - i agree that having both in one methods feels a bit overloaded, but also see the argument for the API usage being redundant here.

what do you think of using Symfony's `ParameterBag` as a blueprint here?
https://github.com/symfony/http-foundation/blob/7.2/ParameterBag.php#L60-L81

Anyhow, not a blocker for this PR and could easily be a follow up - i think the most important part here is the introduction :muscle: 
created_at: 2025-04-25T08:20:22Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,19 @@
+<?php
+
+declare(strict_types=1);
+
+namespace PhpLlm\LlmChain\Model\Response\Metadata;
+
+trait MetadataAwareTrait
+{
+    private ?Metadata $metadata = null;
+
+    public function getMetadata(): Metadata
+    {
+        if (null === $this->metadata) {
+            $this->metadata = new Metadata();
+        }
+
+        return $this->metadata;
path: src/Model/Response/Metadata/MetadataAwareTrait.php
user: login: chr-hertel
body: could be this?
```suggestion
        return $this->metadata ??= new Metadata();
```
created_at: 2025-04-25T08:28:28Z
author_association: MEMBER
diff_hunk: @@ -27,10 +31,27 @@ public function getContent(): string|iterable|object|null
         return $this->unwrap()->getContent();
     }
 
+    public function getRawResponse(): HttpResponse
+    {
+        return $this->response;
+    }
+
+    public function setRawResponse(HttpResponse $rawResponse): void
+    {
+        // Empty by design as the raw response is already set in the constructor and must only be set once
+        throw new RawResponseAlreadySet();
+    }
+
     public function unwrap(): ResponseInterface
     {
         if (!$this->isConverted) {
             $this->convertedResponse = $this->responseConverter->convert($this->response, $this->options);
+
+            if (null === $this->convertedResponse->getRawResponse()) {
+                // Fallback to set the raw response when it was not handled by the response converter itself
+                $this->convertedResponse->setRawResponse($this->response);
path: src/Model/Response/AsyncResponse.php
user: login: chr-hertel
body: yup, fine, was just curious if i read it right
created_at: 2025-04-25T08:32:02Z
author_association: MEMBER

PR comments:
user: login: OskarStark
created_at: 2025-04-08T11:57:26Z
body: Looks great, I would only vote vor having a `Metadata` object instead of an array
author_association: CONTRIBUTOR
user: login: DZunke
created_at: 2025-04-08T16:02:03Z
body: > Looks great, I would only vote vor having a `Metadata` object instead of an array

Thanks ... 👍🏼 ... i thought about extending the collection, if it is generally fine, with a key value (as object) storage and search by name, etc. so a bit of comfort in the metadata then. Just was unsure if this kind of "transport" is in the interest of the library or if someone has other restructurings in mind that would make it "easier" or more complicated. 
author_association: CONTRIBUTOR
user: login: chr-hertel
created_at: 2025-04-09T17:07:12Z
body: Yeees, we need that - thanks for kicking it off! :heart_eyes:
I also like the two levels here with a more structured metadata object and the raw response.

And I would even prefer to have them lower. Basically in the `ResponseInterface` directly:

```php
namespace PhpLlm\LlmChain\Model\Response;

use PhpLlm\LlmChain\Model\Response\Metadata;
use Symfony\Contracts\HttpClient\ResponseInterface as SymfonyHttpResponse;

interface ResponseInterface
{
    /**
     * @return string|iterable<mixed>|object|null
     */
    public function getContent(): string|iterable|object|null;

    /**
     * Returns a key-value set based on provider specific response meta data.
     */
    public function getMetadata(): Metadata;

    /**
     * Return the original API response.
     */
    public function getRawResponse(): SymfonyHttpResponse;
}
```

I know that this would shift the responsibility from the `OutputProcessor` to every `ResponseConverter`, but maybe we can shift that into more clever abstractions/helper/trait/younameit if there are redundancies.

@4lxndr @tryvin that would also help with your use-cases, right?
author_association: MEMBER
user: login: tryvin
created_at: 2025-04-09T17:16:58Z
body: @chr-hertel Yes, it does! Thanks for letting me know!

I've been so busy dealing with external providers that I didn't have the time to do what I promised in the issue.

But I will keep an eye on this PR
author_association: CONTRIBUTOR
user: login: chr-hertel
created_at: 2025-04-15T20:55:10Z
body: While working on #280 and with that fake PipelineResponse I was wondering if we should rather use terminology like Input/Output on this layer already/as well - instead of Request/Response - what do you think?
author_association: MEMBER
user: login: DZunke
created_at: 2025-04-16T05:07:28Z
body: Would this not very confusing? Then there would be two different "Output" named layers within the data flow. 
author_association: CONTRIBUTOR
user: login: chr-hertel
created_at: 2025-04-16T05:18:40Z
body: Yeah true, having that terminology twice could be annoying 🤔
author_association: MEMBER
user: login: OskarStark
created_at: 2025-04-16T06:22:40Z
body: ModelResponse? ModelOutput? To be more specific regarding the layer? Just thinking out loud 
author_association: CONTRIBUTOR
user: login: chr-hertel
created_at: 2025-04-21T15:23:56Z
body: Nevermind the pipeline if it's only the commit thingy - need to dig deeper why that is so flaky
author_association: MEMBER
user: login: chr-hertel
created_at: 2025-04-22T19:20:13Z
body: from an OOP point of view - why choose traits over an abstract base implementation of the response? isn't it that all response classes have the two traits?

with having the interface people could still decide to do their own - or we could do both - having a base class that uses both traits. not sure if i miss the downside here
author_association: MEMBER
user: login: DZunke
created_at: 2025-04-24T05:32:33Z
body: > from an OOP point of view - why choose traits over an abstract base implementation of the response? isn't it that all response classes have the two traits?
> 
> with having the interface people could still decide to do their own - or we could do both - having a base class that uses both traits. not sure if i miss the downside here

Mhm. Yeah. First i wanted to have a most flexible approach but after revisiting it i have implemented an abstract base response to extend from if needed. But i also kept the traits as a handy helpers, if this is fine? See `AsyncResponse` where i have not extended from the `BaseResponse` but utilized the `MetadataAwareTrait` as it has it's own raw symfony response usage. 

What do you think? 
author_association: CONTRIBUTOR
user: login: OskarStark
created_at: 2025-04-25T10:33:17Z
body: Nice one 👏 
author_association: CONTRIBUTOR