Only parse direction once per direction

Solves an InvalidArgumentException when defining two serializers for one
direction:
```yml
// ..
    handler_callbacks:
        serialization:
            json: serializeToJson
            json_ld: serializeToJsonLd
```