Repository: sashabaranov/go-openai
Pull request #733 (2024-05-05 13:40:23)

Add support for word-level audio transcription timestamp granularity
**Describe the change**
Add support for word-level audio transcription timestamp granularity.

**Provide OpenAI documentation link**
- [timestamp_granularities[]](https://platform.openai.com/docs/api-reference/audio/createTranscription#audio-createtranscription-timestamp_granularities)
- [words](https://platform.openai.com/docs/api-reference/audio/verbose-json-object#audio/verbose-json-object-words)

**Describe your solution**
Added `AudioRequest.TimestampGranularities` and `AudioResponse.Words` fields.

**Tests**
Filled `AudioRequest.TimestampGranularities` field in the existing tests' audio requests.

**Additional context**

- Related to two-months-abandoned #673, self-closed #680, and self-closed #674.
- Also removed some obsolete comments on `AudioRequest` fields.

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/sashabaranov/go-openai/pull/733?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 98.68%. Comparing base [(`774fc9d`)](https://app.codecov.io/gh/sashabaranov/go-openai/commit/774fc9dd12ed60c10a9f9f03319ddb9cd5f8780c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) to head [(`456ceba`)](https://app.codecov.io/gh/sashabaranov/go-openai/pull/733?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None).
> Report is 10 commits behind head on master.


<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##           master     #733      +/-   ##
==========================================
+ Coverage   98.46%   98.68%   +0.22%     
==========================================
  Files          24       24              
  Lines        1364     1140     -224     
==========================================
- Hits         1343     1125     -218     
+ Misses         15        9       -6     
  Partials        6        6              
```



</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/sashabaranov/go-openai/pull/733?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None).

agcom: I would have proposes separating transcription and translation request/response types, but that would have been a breaking change.
sashabaranov: Thank you for working on this!
sashabaranov: @agcom the PR looks good to me in its current state!

> I would have proposed separating transcription and translation request/response types, but that would have been a breaking change.

We can discuss it in a separate PR, also totally open to look at a sketch of how this might look like 🙌🏻 
agcom: @sashabaranov alright then, ready to review. I just wanted to test it out in our development server before marking it for review (it works fine).

> We can discuss it in a separate PR, also totally open to look at a sketch of how this might look like 🙌🏻

May god bless me with more code refactoring tasks so I would work on this :smile:.

Patch:
diff --git a/audio.go b/audio.go
index 4cbe4fe..dbc26d1 100644
--- a/audio.go
+++ b/audio.go
@@ -27,8 +27,14 @@ const (
 	AudioResponseFormatVTT         AudioResponseFormat = "vtt"
 )
 
+type TranscriptionTimestampGranularity string
+
+const (
+	TranscriptionTimestampGranularityWord    TranscriptionTimestampGranularity = "word"
+	TranscriptionTimestampGranularitySegment TranscriptionTimestampGranularity = "segment"
+)
+
 // AudioRequest represents a request structure for audio API.
-// ResponseFormat is not supported for now. We only return JSON text, which may be sufficient.
 type AudioRequest struct {
 	Model string
 
@@ -38,10 +44,11 @@ type AudioRequest struct {
 	// Reader is an optional io.Reader when you do not want to use an existing file.
 	Reader io.Reader
 
-	Prompt      string // For translation, it should be in English
-	Temperature float32
-	Language    string // For translation, just do not use it. It seems "en" works, not confirmed...
-	Format      AudioResponseFormat
+	Prompt                 string
+	Temperature            float32
+	Language               string // Only for transcription.
+	Format                 AudioResponseFormat
+	TimestampGranularities []TranscriptionTimestampGranularity // Only for transcription.
 }
 
 // AudioResponse represents a response structure for audio API.
@@ -62,6 +69,11 @@ type AudioResponse struct {
 		NoSpeechProb     float64 `json:"no_speech_prob"`
 		Transient        bool    `json:"transient"`
 	} `json:"segments"`
+	Words []struct {
+		Word  string  `json:"word"`
+		Start float64 `json:"start"`
+		End   float64 `json:"end"`
+	} `json:"words"`
 	Text string `json:"text"`
 
 	httpHeader
@@ -179,6 +191,15 @@ func audioMultipartForm(request AudioRequest, b utils.FormBuilder) error {
 		}
 	}
 
+	if len(request.TimestampGranularities) > 0 {
+		for _, tg := range request.TimestampGranularities {
+			err = b.WriteField("timestamp_granularities[]", string(tg))
+			if err != nil {
+				return fmt.Errorf("writing timestamp_granularities[]: %w", err)
+			}
+		}
+	}
+
 	// Close the multipart writer
 	return b.Close()
 }


Test patch:
diff --git a/audio_api_test.go b/audio_api_test.go
index a0efc79..c245984 100644
--- a/audio_api_test.go
+++ b/audio_api_test.go
@@ -105,6 +105,10 @@ func TestAudioWithOptionalArgs(t *testing.T) {
 				Temperature: 0.5,
 				Language:    "zh",
 				Format:      openai.AudioResponseFormatSRT,
+				TimestampGranularities: []openai.TranscriptionTimestampGranularity{
+					openai.TranscriptionTimestampGranularitySegment,
+					openai.TranscriptionTimestampGranularityWord,
+				},
 			}
 			_, err := tc.createFn(ctx, req)
 			checks.NoError(t, err, "audio API error")
diff --git a/audio_test.go b/audio_test.go
index 5346244..235931f 100644
--- a/audio_test.go
+++ b/audio_test.go
@@ -24,6 +24,10 @@ func TestAudioWithFailingFormBuilder(t *testing.T) {
 		Temperature: 0.5,
 		Language:    "en",
 		Format:      AudioResponseFormatSRT,
+		TimestampGranularities: []TranscriptionTimestampGranularity{
+			TranscriptionTimestampGranularitySegment,
+			TranscriptionTimestampGranularityWord,
+		},
 	}
 
 	mockFailedErr := fmt.Errorf("mock form builder fail")
@@ -47,7 +51,7 @@ func TestAudioWithFailingFormBuilder(t *testing.T) {
 		return nil
 	}
 
-	failOn := []string{"model", "prompt", "temperature", "language", "response_format"}
+	failOn := []string{"model", "prompt", "temperature", "language", "response_format", "timestamp_granularities[]"}
 	for _, failingField := range failOn {
 		failForField = failingField
 		mockFailedErr = fmt.Errorf("mock form builder fail on field %s", failingField)