Repository: airbytehq/airbyte
Pull request #55229: (feat: Salesloft) - Add emails_scoped_fields stream
Author: login: tybernstein
Labels: name: area/documentation
description: Improvements or additions to documentation
name: area/connectors
description: Connector related issues
name: connectors/source/salesloft
Created: 2025-03-06T15:03:40Z | Merged: 2025-03-10T17:06:51Z

Adds a new stream which is the same as the emails stream with the addition of the scoped_fields: body and subject. These fields require a specific Salesloft plan and are not available to everyone which is why they are on their own stream. Removes body and subject fields from `emails` stream schema as they are null without the required parameters added to the request.

## What
Fixes https://github.com/airbytehq/airbyte/issues/55221

## How
Adds a new stream with the required query parameter scoped_fields[]: ["subject","body"]

## Review guide
<!--
1. `x.py`
2. `y.py`
-->

## User Impact
By using a new dedicated stream it should not impact existing users. Potential negative impact is that the new stream is only available for specific Salesloft plans

## Can this PR be safely reverted and rolled back?
<!--
* If unsure, leave it blank.
-->
- [ ] YES 💚
- [ ] NO ❌



Diff:
diff --git a/airbyte-integrations/connectors/source-salesloft/manifest.yaml b/airbyte-integrations/connectors/source-salesloft/manifest.yaml
index 8d13e32a7726..277763dc9113 100644
--- a/airbyte-integrations/connectors/source-salesloft/manifest.yaml
+++ b/airbyte-integrations/connectors/source-salesloft/manifest.yaml
@@ -827,6 +827,83 @@ definitions:
         type: InlineSchemaLoader
         schema:
           $ref: "#/schemas/emails"
+    emails_scoped_fields:
+      type: DeclarativeStream
+      name: emails_scoped_fields
+      primary_key:
+        - id
+      retriever:
+        type: SimpleRetriever
+        requester:
+          $ref: "#/definitions/base_requester"
+          path: /activities/emails
+          http_method: GET
+          request_parameters:
+            sort_direction: ASC
+            scoped_fields[]: "[\"subject\",\"body\"]"
+          error_handler:
+            type: CompositeErrorHandler
+            error_handlers:
+              - type: DefaultErrorHandler
+                response_filters:
+                  - type: HttpResponseFilter
+                    action: FAIL
+                    error_message_contains: required scope access not included
+                    error_message: >-
+                      Scoped Fields require a Premier Salesloft Plan - Requested
+                      scoped field(s) could not be returned, required scope
+                      access not included
+        record_selector:
+          type: RecordSelector
+          extractor:
+            type: DpathExtractor
+            field_path:
+              - data
+        paginator:
+          type: DefaultPaginator
+          page_token_option:
+            type: RequestOption
+            inject_into: request_parameter
+            field_name: page
+          page_size_option:
+            type: RequestOption
+            field_name: per_page
+            inject_into: request_parameter
+          pagination_strategy:
+            type: CursorPagination
+            page_size: 100
+            cursor_value: "{{ response['metadata']['paging']['next_page'] }}"
+            stop_condition: >-
+              {{ response['metadata']['paging'] is not defined or not
+              response['metadata']['paging'] or
+              response['metadata']['paging']['next_page'] is not defined or not
+              response['metadata']['paging']['next_page'] }}
+      incremental_sync:
+        type: DatetimeBasedCursor
+        cursor_field: updated_at
+        cursor_datetime_formats:
+          - "%Y-%m-%dT%H:%M:%S.%f%z"
+        datetime_format: "%Y-%m-%dT%H:%M:%S.%fZ"
+        start_datetime:
+          type: MinMaxDatetime
+          datetime: "{{ config['start_date'] }}"
+          datetime_format: "%Y-%m-%dT%H:%M:%SZ"
+        start_time_option:
+          type: RequestOption
+          inject_into: request_parameter
+          field_name: updated_at[gte]
+        end_time_option:
+          type: RequestOption
+          inject_into: request_parameter
+          field_name: updated_at[lte]
+        end_datetime:
+          type: MinMaxDatetime
+          datetime: "{{ now_utc().strftime('%Y-%m-%dT%H:%M:%SZ') }}"
+          datetime_format: "%Y-%m-%dT%H:%M:%SZ"
+      schema_loader:
+        type: InlineSchemaLoader
+        schema:
+          $ref: "#/schemas/emails_scoped_fields"
     calls:
       type: DeclarativeStream
       name: calls
@@ -1561,6 +1638,7 @@ streams:
   - $ref: "#/definitions/streams/cadences"
   - $ref: "#/definitions/streams/cadence_memberships"
   - $ref: "#/definitions/streams/emails"
+  - $ref: "#/definitions/streams/emails_scoped_fields"
   - $ref: "#/definitions/streams/calls"
   - $ref: "#/definitions/streams/accounts"
   - $ref: "#/definitions/streams/account_stages"
@@ -1673,6 +1751,7 @@ metadata:
     cadences: true
     cadence_memberships: true
     emails: true
+    emails_scoped_fields: false
     calls: true
     accounts: true
     account_stages: true
@@ -3083,14 +3162,6 @@ schemas:
         type:
           - "null"
           - boolean
-      subject:
-        type:
-          - "null"
-          - string
-      body:
-        type:
-          - "null"
-          - string
       cadence:
         type:
           - "null"
@@ -3283,6 +3354,147 @@ schemas:
         type:
           - "null"
           - boolean
+  emails_scoped_fields:
+    type: object
+    $schema: http://json-schema.org/schema#
+    additionalProperties: true
+    properties:
+      additional_recipients:
+        type:
+          - array
+          - "null"
+      body:
+        type:
+          - string
+          - "null"
+      bounced:
+        type:
+          - boolean
+          - "null"
+      click_tracking:
+        type:
+          - boolean
+          - "null"
+      counts:
+        type:
+          - object
+          - "null"
+        properties:
+          attachments:
+            type:
+              - number
+              - "null"
+          clicks:
+            type:
+              - number
+              - "null"
+          replies:
+            type:
+              - number
+              - "null"
+          unique_devices:
+            type:
+              - number
+              - "null"
+          unique_locations:
+            type:
+              - number
+              - "null"
+          views:
+            type:
+              - number
+              - "null"
+      created_at:
+        type:
+          - string
+          - "null"
+      crm_activity:
+        type:
+          - object
+          - "null"
+        properties:
+          _href:
+            type:
+              - string
+              - "null"
+          id:
+            type:
+              - number
+              - "null"
+      headers:
+        type:
+          - object
+          - "null"
+      id:
+        type: number
+      mailing:
+        type:
+          - object
+          - "null"
+        properties:
+          id:
+            type:
+              - number
+              - "null"
+      personalization:
+        type:
+          - string
+          - "null"
+      recipient:
+        type:
+          - object
+          - "null"
+        properties:
+          _href:
+            type:
+              - string
+              - "null"
+          id:
+            type:
+              - number
+              - "null"
+      recipient_email_address:
+        type:
+          - string
+          - "null"
+      send_after:
+        type:
+          - string
+          - "null"
+      sent_at:
+        type:
+          - string
+          - "null"
+      status:
+        type:
+          - string
+          - "null"
+      subject:
+        type:
+          - string
+          - "null"
+      updated_at:
+        type: string
+      user:
+        type:
+          - object
+          - "null"
+        properties:
+          _href:
+            type:
+              - string
+              - "null"
+          id:
+            type:
+              - number
+              - "null"
+      view_tracking:
+        type:
+          - boolean
+          - "null"
+    required:
+      - id
+      - updated_at
   calls:
     type: object
     $schema: http://json-schema.org/draft-07/schema#
diff --git a/airbyte-integrations/connectors/source-salesloft/metadata.yaml b/airbyte-integrations/connectors/source-salesloft/metadata.yaml
index b0731a2d4643..f96e9e048203 100644
--- a/airbyte-integrations/connectors/source-salesloft/metadata.yaml
+++ b/airbyte-integrations/connectors/source-salesloft/metadata.yaml
@@ -22,7 +22,7 @@ data:
   connectorSubtype: api
   connectorType: source
   definitionId: 41991d12-d4b5-439e-afd0-260a31d4c53f
-  dockerImageTag: 1.4.0
+  dockerImageTag: 1.5.0
   dockerRepository: airbyte/source-salesloft
   githubIssueLabel: source-salesloft
   icon: icon.svg
diff --git a/docs/integrations/sources/salesloft.md b/docs/integrations/sources/salesloft.md
index 2a149ac2b4b4..9e8ee39e71f3 100644
--- a/docs/integrations/sources/salesloft.md
+++ b/docs/integrations/sources/salesloft.md
@@ -69,6 +69,7 @@ This connector outputs the following streams:
 - [People](https://developers.salesloft.com/api.html#!/People/get_v2_people_json)
 - [Users](https://developers.salesloft.com/api.html#!/Users/get_v2_users_json)
 - [Emails](https://developers.salesloft.com/api.html#!/Emails/get_v2_activities_emails_json)
+- [Emails Scoped Fields](https://developers.salesloft.com/api.html#!/Emails/get_v2_activities_emails_json) **Warning:** This stream requires a Premier Salesloft Plan
 - [Account Stages](https://developers.salesloft.com/api.html#!/Account_Stages/get_v2_account_stages_json)
 - [Account Tiers](https://developers.salesloft.com/api.html#!/Account_Tiers/get_v2_account_tiers_json)
 - [Accounts](https://developers.salesloft.com/api.html#!/Accounts/get_v2_accounts_json)
@@ -105,6 +106,7 @@ Salesloft has the [rate limits](hhttps://developers.salesloft.com/api.html#!/Top
 
 | Version | Date       | Pull Request                                             | Subject                                                        |
 | :------ | :--------- | :------------------------------------------------------- | :------------------------------------------------------------- |
+| 1.5.0 | 2025-03-06 | [55229](https://github.com/airbytehq/airbyte/pull/55229) | Add emails_scoped_fields stream. Remove body and subject fields from email stream |
 | 1.4.0 | 2025-02-18 | [54136](https://github.com/airbytehq/airbyte/pull/54136) | add body and subject fields to email stream |
 | 1.3.1 | 2024-12-03 | [48770](https://github.com/airbytehq/airbyte/pull/48770) | Remove duplication in manifest, fix query param updated_at[gt] |
 | 1.3.0 | 2024-11-04 | [47298](https://github.com/airbytehq/airbyte/pull/47298) | Migrate to manifest only format |


Reviews:
user: login: agarctfi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T15:41:27Z
user: login: agarctfi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T15:42:24Z
user: login: agarctfi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T15:43:37Z
user: login: agarctfi
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T15:45:04Z
user: login: natikgadzhi
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T17:06:42Z

Review comments:
diff_hunk: @@ -3083,14 +3162,6 @@ schemas:
         type:
           - "null"
           - boolean
-      subject:
path: airbyte-integrations/connectors/source-salesloft/manifest.yaml
user: login: agarctfi
body: These are removed because the current `emails` stream does not have the added `request_parameters`:
`'scoped_fields[]': '["subject","body"]'` - which tell the API to populate these fields. 

So we are just removing null fields, which seems fine to me.
created_at: 2025-03-06T15:41:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -3283,6 +3354,147 @@ schemas:
         type:
           - "null"
           - boolean
+  emails_scoped_fields:
path: airbyte-integrations/connectors/source-salesloft/manifest.yaml
user: login: agarctfi
body: Copy of the `emails` stream but with the added schema `subject` and `body`
created_at: 2025-03-06T15:42:23Z
author_association: CONTRIBUTOR
diff_hunk: @@ -827,6 +827,83 @@ definitions:
         type: InlineSchemaLoader
         schema:
           $ref: "#/schemas/emails"
+    emails_scoped_fields:
+      type: DeclarativeStream
+      name: emails_scoped_fields
+      primary_key:
+        - id
+      retriever:
+        type: SimpleRetriever
+        requester:
+          $ref: "#/definitions/base_requester"
+          path: /activities/emails
+          http_method: GET
+          request_parameters:
+            sort_direction: ASC
+            scoped_fields[]: "[\"subject\",\"body\"]"
path: airbyte-integrations/connectors/source-salesloft/manifest.yaml
user: login: agarctfi
body: Added parameters to populate these fields. 
created_at: 2025-03-06T15:43:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -827,6 +827,83 @@ definitions:
         type: InlineSchemaLoader
         schema:
           $ref: "#/schemas/emails"
+    emails_scoped_fields:
+      type: DeclarativeStream
+      name: emails_scoped_fields
+      primary_key:
+        - id
+      retriever:
+        type: SimpleRetriever
+        requester:
+          $ref: "#/definitions/base_requester"
+          path: /activities/emails
+          http_method: GET
+          request_parameters:
+            sort_direction: ASC
+            scoped_fields[]: "[\"subject\",\"body\"]"
+          error_handler:
+            type: CompositeErrorHandler
+            error_handlers:
+              - type: DefaultErrorHandler
+                response_filters:
+                  - type: HttpResponseFilter
+                    action: FAIL
+                    error_message_contains: required scope access not included
+                    error_message: >-
+                      Scoped Fields require a Premier Salesloft Plan - Requested
+                      scoped field(s) could not be returned, required scope
+                      access not included
path: airbyte-integrations/connectors/source-salesloft/manifest.yaml
user: login: agarctfi
body: Error handler for users that do not have the premium plan to request these fields. 
created_at: 2025-03-06T15:45:04Z
author_association: CONTRIBUTOR

PR comments:
user: login: vercel[bot]
created_at: 2025-03-06T15:03:45Z
body: [vc]: #K5b2CBzFPtC651M5weIL5mHQexSD5skhf9fPLXJhnmg=:eyJpc01vbm9yZXBvIjp0cnVlLCJ0eXBlIjoiZ2l0aHViIiwicHJvamVjdHMiOlt7Im5hbWUiOiJhaXJieXRlLWRvY3MiLCJsaXZlRmVlZGJhY2siOnsicmVzb2x2ZWQiOjAsInVucmVzb2x2ZWQiOjAsInRvdGFsIjowLCJsaW5rIjoiYWlyYnl0ZS1kb2NzLWdpdC10eWJlcm5zdGVpbi1wYXRjaC0xLWFpcmJ5dGUtZ3Jvd3RoLnZlcmNlbC5hcHAifSwiaW5zcGVjdG9yVXJsIjoiaHR0cHM6Ly92ZXJjZWwuY29tL2FpcmJ5dGUtZ3Jvd3RoL2FpcmJ5dGUtZG9jcy85VjRDRHdRcU1QdnY5TXZRQzZyckd3WWNGNllnIiwicHJldmlld1VybCI6ImFpcmJ5dGUtZG9jcy1naXQtdHliZXJuc3RlaW4tcGF0Y2gtMS1haXJieXRlLWdyb3d0aC52ZXJjZWwuYXBwIiwibmV4dENvbW1pdFN0YXR1cyI6IkRFUExPWUVEIiwicm9vdERpcmVjdG9yeSI6ImRvY3VzYXVydXMifV19
**The latest updates on your projects**. Learn more about [Vercel for Git ↗︎](https://vercel.link/github-learn-more)

| Name | Status | Preview | Comments | Updated (UTC) |
| :--- | :----- | :------ | :------- | :------ |
| **airbyte-docs** | ✅ Ready ([Inspect](https://vercel.com/airbyte-growth/airbyte-docs/9V4CDwQqMPvv9MvQC6rrGwYcF6Yg)) | [Visit Preview](https://airbyte-docs-git-tybernstein-patch-1-airbyte-growth.vercel.app) | 💬 [**Add feedback**](https://vercel.live/open-feedback/airbyte-docs-git-tybernstein-patch-1-airbyte-growth.vercel.app?via=pr-comment-feedback-link) | Mar 6, 2025 3:26pm |


author_association: NONE
user: login: agarctfi
created_at: 2025-03-06T15:49:17Z
body: @airbytehq/dev-extensibility I added my comments, but I think this looks good!
author_association: CONTRIBUTOR