Repository: amp-labs/connectors
Pull request #1270 (2025-01-08 22:49:33)

[CON-967] Hubpsot with FieldsMetadata
# Changes

1. Defined `FieldMetadataMap` on `ObjectMetadata`.
2. Marked `FieldsMap` as deprecated (will be reflected in IDE). `ObjectMetadata` constructor will populate this field automatically as a derivative of `FieldMetadataMap`.
3. Hubspot has a method that transforms data acquired from provider API (the gist of the PR).
4. Extended mock tests which make extended use of properties. Note: the sample response has a bit of everything, ex: string, boolean, enumaration, etc. Repetitive fields are truncated for brevity, real endpoint response is just enormous.

# Review
Please refer to the mock tests to understand the output, given that provider returns this JSON: https://github.com/amp-labs/connectors/blob/cobalt0s/hubspot-value-type/providers/hubspot/test/metadata-contacts-properties-sampled.json

Cobalt0s: > [!WARNING]
> <b>This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack <a href="https://app.graphite.dev/github/pr/amp-labs/connectors/1270?utm_source=stack-comment-downstack-mergeability-warning" >on Graphite</a>.</b>
> <a href="https://graphite.dev/docs/merge-pull-requests">Learn more</a>

* **#1270** <a href="https://app.graphite.dev/github/pr/amp-labs/connectors/1270?utm_source=stack-comment-icon" target="_blank"><img src="https://static.graphite.dev/graphite-32x32-black.png" alt="Graphite" width="10px" height="10px"/></a> 👈 <a href="https://app.graphite.dev/github/pr/amp-labs/connectors/1270?utm_source=stack-comment-view-in-graphite" target="_blank">(View in Graphite)</a>
* **#1269** <a href="https://app.graphite.dev/github/pr/amp-labs/connectors/1269?utm_source=stack-comment-icon" target="_blank"><img src="https://static.graphite.dev/graphite-32x32-black.png" alt="Graphite" width="10px" height="10px"/></a>
* `main`


<h2></h2>

This stack of pull requests is managed by <a href="https://graphite.dev?utm-source=stack-comment"><b>Graphite</b></a>. Learn more about <a href="https://stacking.dev/?utm_source=stack-comment">stacking</a>.
<!-- Current dependencies on/for this PR: -->
laurenzlong: @Cobalt0s I realized that we don't actually have an end to end test for Hubspot's ListObjectMetadata method, can you extend test/hubspot/metadata/main.go to also call ListObjectMetadata and dump out the result? And then add the screenshot to this PR?
Cobalt0s: > @Cobalt0s I realized that we don't actually have an end to end test for Hubspot's ListObjectMetadata method, can you extend test/hubspot/metadata/main.go to also call ListObjectMetadata and dump out the result? And then add the screenshot to this PR?

It is there in Graphite stack, please have a look at https://github.com/amp-labs/connectors/pull/1269.

I deliberitly created 2 PRs, first is adding `manual+mock` tests. Let me know if you don't like the "manual test". I find it confusing to this day that the repo is called a test. It is something in between examples and real time tests.



Patch:
diff --git a/common/types.go b/common/types.go
index 426fa106..aa27aba1 100644
--- a/common/types.go
+++ b/common/types.go
@@ -256,13 +256,52 @@ type ListObjectMetadataResult struct {
 }
 
 type ObjectMetadata struct {
-	// Provider's display name for the object
+	// Provider's display name for the object.
 	DisplayName string
 
-	// FieldsMap is a map of field names to field display names
+	// Fields is a map of field names to FieldMetadata.
+	Fields map[string]FieldMetadata
+
+	// FieldsMap is a map of field names to field display names.
+	// Deprecated: this map includes only display names.
+	// Refer to Fields for extended description of field properties.
 	FieldsMap map[string]string
 }
 
+// NewObjectMetadata constructs ObjectMetadata.
+// This will automatically infer fields map from field metadata map. This construct exists for such convenience.
+func NewObjectMetadata(displayName string, fields map[string]FieldMetadata) *ObjectMetadata {
+	return &ObjectMetadata{
+		DisplayName: displayName,
+		Fields:      fields,
+		FieldsMap:   inferDeprecatedFieldsMap(fields),
+	}
+}
+
+type FieldMetadata struct {
+	// DisplayName is a human-readable field name.
+	DisplayName string
+
+	// ValueType is a set of Ampersand defined field types.
+	ValueType ValueType
+
+	// ProviderType is the raw type, a term used by provider API.
+	// Each is mapped to an Ampersand ValueType.
+	ProviderType string
+
+	// ReadOnly would indicate if field can be modified or only read.
+	ReadOnly bool
+
+	// Values is a list of possible values for this field.
+	// It is applicable only if the type is either singleSelect or multiSelect, otherwise slice is nil.
+	Values []FieldValue
+}
+
+type FieldValue struct {
+	Value        string
+	DisplayValue string
+}
+
 type PostAuthInfo struct {
 	CatalogVars          *map[string]string
 	RawResponse          *JSONHTTPResponse
@@ -298,3 +337,13 @@ type WebhookVerificationParameters struct {
 	ClientSecret string
 	Method       string
 }
+
+func inferDeprecatedFieldsMap(fields map[string]FieldMetadata) map[string]string {
+	fieldsMap := make(map[string]string)
+
+	for name, field := range fields {
+		fieldsMap[name] = field.DisplayName
+	}
+
+	return fieldsMap
+}
diff --git a/common/valueType.go b/common/valueType.go
new file mode 100644
index 00000000..e3842dd1
--- /dev/null
+++ b/common/valueType.go
@@ -0,0 +1,18 @@
+package common
+
+type ValueType string
+
+const (
+	ValueTypeString  = "string"
+	ValueTypeBoolean = "boolean"
+	ValueTypeFloat   = "float" // float is more preferred than int if provider doesn't differentiate.
+	ValueTypeInt     = "int"
+
+	ValueTypeDate     = "date"
+	ValueTypeDateTime = "datetime"
+
+	ValueTypeSingleSelect = "singleSelect"
+	ValueTypeMultiSelect  = "multiSelect"
+
+	ValueTypeOther = "other"
+)
diff --git a/providers/hubspot/metadata.go b/providers/hubspot/metadata.go
index f0b9f19d..4864582c 100644
--- a/providers/hubspot/metadata.go
+++ b/providers/hubspot/metadata.go
@@ -2,14 +2,13 @@ package hubspot
 
 import (
 	"context"
-	"encoding/json"
 	"fmt"
 	"strconv"
 	"strings"
+	"time"
 
 	"github.com/amp-labs/connectors/common"
 	"github.com/amp-labs/connectors/common/logging"
-	"github.com/spyzhov/ajson"
 )
 
 type objectMetadataResult struct {
@@ -78,15 +77,6 @@ func (c *Connector) ListObjectMetadata( // nolint:cyclop,funlen
 	return objectsMap, nil
 }
 
-type describeObjectResponse struct {
-	Results []describeObjectResult `json:"results"`
-}
-
-type describeObjectResult struct {
-	Name  string `json:"name"`
-	Label string `json:"label"`
-}
-
 // describeObject returns object metadata for the given object name.
 func (c *Connector) describeObject(ctx context.Context, objectName string) (*common.ObjectMetadata, error) {
 	relativeURL := strings.Join([]string{"properties", objectName}, "/")
@@ -101,41 +91,14 @@ func (c *Connector) describeObject(ctx context.Context, objectName string) (*com
 		return nil, fmt.Errorf("error fetching HubSpot fields: %w", err)
 	}
 
-	body, ok := rsp.Body()
-	if !ok {
-		return nil, fmt.Errorf("cannot get HubSpot fields %w", common.ErrEmptyJSONHTTPResponse)
-	}
-
-	rawResponse, err := ajson.Marshal(body)
-	if err != nil {
-		return nil, fmt.Errorf("error marshalling object metadata response into byte array: %w", err)
-	}
-
-	resp := &describeObjectResponse{}
-
-	err = json.Unmarshal(rawResponse, resp)
+	resp, err := common.UnmarshalJSON[fieldDescriptionResponse](rsp)
 	if err != nil {
 		return nil, fmt.Errorf("error unmarshalling object metadata response into JSON: %w", err)
 	}
 
-	return &common.ObjectMetadata{
-		DisplayName: objectName,
-		FieldsMap:   makeFieldsMap(resp),
-	}, nil
-}
-
-// makeFieldsMap returns a map of field name to field label.
-func makeFieldsMap(data *describeObjectResponse) map[string]string {
-	fieldsMap := make(map[string]string)
-
-	for _, field := range data.Results {
-		fieldName := strings.ToLower(field.Name)
-
-		// Add entry to fieldsMap
-		fieldsMap[fieldName] = field.Label
-	}
-
-	return fieldsMap
+	return common.NewObjectMetadata(
+		objectName, resp.transformToFields(),
+	), nil
 }
 
 func (c *Connector) GetPostAuthInfo(
@@ -178,3 +141,118 @@ func (c *Connector) GetAccountInfo(ctx context.Context) (*AccountInfo, *common.J
 
 	return accountInfo, resp, nil
 }
+
+type fieldDescriptionResponse struct {
+	Results []fieldDescription `json:"results"`
+}
+
+type fieldDescription struct {
+	UpdatedAt            time.Time                 `json:"updatedAt"`
+	CreatedAt            time.Time                 `json:"createdAt"`
+	Name                 string                    `json:"name"`
+	Label                string                    `json:"label"`
+	Type                 string                    `json:"type"`
+	FieldType            string                    `json:"fieldType"`
+	Description          string                    `json:"description"`
+	GroupName            string                    `json:"groupName"`
+	Options              []fieldEnumerationOption  `json:"options"`
+	DisplayOrder         int                       `json:"displayOrder"`
+	Calculated           bool                      `json:"calculated"`
+	ExternalOptions      bool                      `json:"externalOptions"`
+	HasUniqueValue       bool                      `json:"hasUniqueValue"`
+	Hidden               bool                      `json:"hidden"`
+	HubspotDefined       bool                      `json:"hubspotDefined"`
+	ModificationMetadata fieldModificationMetadata `json:"modificationMetadata"`
+	FormField            bool                      `json:"formField"`
+	DataSensitivity      string                    `json:"dataSensitivity"`
+}
+
+type fieldEnumerationOption struct {
+	Label        string `json:"label"`
+	Value        string `json:"value"`
+	Description  string `json:"description"`
+	DisplayOrder int    `json:"displayOrder"`
+	Hidden       bool   `json:"hidden"`
+}
+
+type fieldModificationMetadata struct {
+	Archivable         bool `json:"archivable"`
+	ReadOnlyDefinition bool `json:"readOnlyDefinition"`
+	ReadOnlyValue      bool `json:"readOnlyValue"`
+}
+
+func (r fieldDescriptionResponse) transformToFields() map[string]common.FieldMetadata {
+	fieldsMap := make(map[string]common.FieldMetadata)
+
+	for _, field := range r.Results {
+		fieldName := strings.ToLower(field.Name)
+		fieldsMap[fieldName] = field.transformToFieldMetadata()
+	}
+
+	return fieldsMap
+}
+
+// transformToFieldMetadata converts Provider model of a field into Ampersand's common.FieldMetadata.
+// This normalizes provider response to the unified standard across all providers.
+func (o fieldDescription) transformToFieldMetadata() common.FieldMetadata {
+	var (
+		valueType common.ValueType
+		values    []common.FieldValue
+	)
+
+	// Based on type and field type properties from Hubspot object model map value to Ampersand value type.
+	switch o.Type {
+	case "string":
+		valueType = common.ValueTypeString
+	case "number":
+		valueType = common.ValueTypeFloat
+	case "bool":
+		valueType = common.ValueTypeBoolean
+	case "datetime":
+		valueType = common.ValueTypeDateTime
+	case "enumeration":
+		valueType, values = o.implyEnumerationType()
+		// Enumeration type means there are predefined field values.
+	default:
+		// ex: object_coordinates, phone_number
+		valueType = common.ValueTypeOther
+	}
+
+	return common.FieldMetadata{
+		DisplayName:  o.Label,
+		ValueType:    valueType,
+		ProviderType: o.Type + "." + o.FieldType,
+		ReadOnly:     o.ModificationMetadata.ReadOnlyValue,
+		Values:       values,
+	}
+}
+
+func (o fieldDescription) implyEnumerationType() (common.ValueType, []common.FieldValue) {
+	var values []common.FieldValue
+
+	if len(o.Options) != 0 {
+		// List of values is not nil, at least one option exists.
+		values = make([]common.FieldValue, len(o.Options))
+		for index, option := range o.Options {
+			values[index] = common.FieldValue{
+				Value:        option.Value,
+				DisplayValue: option.Label,
+			}
+		}
+	}
+
+	switch o.FieldType {
+	case "checkbox":
+		return common.ValueTypeMultiSelect, values
+	case "booleancheckbox":
+		// Boolean values are ignored.
+		return common.ValueTypeBoolean, nil
+	case "radio":
+		return common.ValueTypeSingleSelect, values
+	case "select":
+		return common.ValueTypeSingleSelect, values
+	default:
+		// ex: enumeration.calculation_equation
+		return common.ValueTypeOther, values
+	}
+}


Test patch:
diff --git a/providers/hubspot/metadata_test.go b/providers/hubspot/metadata_test.go
index 16282c69..7ef18f78 100644
--- a/providers/hubspot/metadata_test.go
+++ b/providers/hubspot/metadata_test.go
@@ -37,6 +37,128 @@ func TestListObjectMetadata(t *testing.T) { // nolint:funlen,gocognit,cyclop
 				Result: map[string]common.ObjectMetadata{
 					"contacts": {
 						DisplayName: "contacts",
+						Fields: map[string]common.FieldMetadata{
+							// String
+							"address": {
+								DisplayName:  "Street Address",
+								ValueType:    "string",
+								ProviderType: "string.text",
+								ReadOnly:     false,
+								Values:       nil,
+							},
+							"mobilephone": {
+								DisplayName:  "Mobile Phone Number",
+								ValueType:    "string",
+								ProviderType: "string.phonenumber",
+								ReadOnly:     false,
+								Values:       nil,
+							},
+
+							// Boolean.
+							"hs_contact_enrichment_opt_out": {
+								DisplayName:  "Enrichment opt out",
+								ValueType:    "boolean",
+								ProviderType: "bool.booleancheckbox",
+								ReadOnly:     true,
+								Values:       nil,
+							},
+							"autogen": {
+								DisplayName:  "autogen",
+								ValueType:    "boolean",
+								ProviderType: "enumeration.booleancheckbox",
+								ReadOnly:     false,
+								Values:       nil,
+							},
+
+							// Float.
+							"associatedcompanyid": {
+								DisplayName:  "Primary Associated Company ID",
+								ValueType:    "float",
+								ProviderType: "number.number",
+								ReadOnly:     false,
+								Values:       nil,
+							},
+							"hubspotscore": {
+								DisplayName:  "HubSpot Score",
+								ValueType:    "float",
+								ProviderType: "number.calculation_score",
+								ReadOnly:     true,
+								Values:       nil,
+							},
+							"hs_associated_target_accounts": {
+								DisplayName:  "Associated Target Accounts",
+								ValueType:    "float",
+								ProviderType: "number.calculation_rollup",
+								ReadOnly:     true,
+								Values:       nil,
+							},
+
+							// Single/Multi Select.
+							"hs_content_membership_status": {
+								DisplayName:  "Status",
+								ValueType:    "singleSelect",
+								ProviderType: "enumeration.select",
+								ReadOnly:     false,
+								Values: []common.FieldValue{{
+									Value:        "active",
+									DisplayValue: "Active",
+								}, {
+									Value:        "inactive",
+									DisplayValue: "Inactive",
+								}},
+							},
+							"hs_predictivecontactscorebucket": {
+								DisplayName:  "Lead Rating",
+								ValueType:    "singleSelect",
+								ProviderType: "enumeration.radio",
+								ReadOnly:     true,
+								Values: []common.FieldValue{{
+									Value:        "bucket_1",
+									DisplayValue: "1 Star",
+								}, {
+									Value:        "bucket_2",
+									DisplayValue: "2 Stars",
+								}, {
+									Value:        "bucket_3",
+									DisplayValue: "3 Stars",
+								}, {
+									Value:        "bucket_4",
+									DisplayValue: "4 Stars",
+								}},
+							},
+							"hs_all_assigned_business_unit_ids": {
+								DisplayName:  "Business units",
+								ValueType:    "multiSelect",
+								ProviderType: "enumeration.checkbox",
+								ReadOnly:     false,
+								Values:       nil,
+							},
+
+							// Datetime.
+							"hs_first_subscription_create_date": {
+								DisplayName:  "First subscription create date",
+								ValueType:    "datetime",
+								ProviderType: "datetime.calculation_rollup",
+								ReadOnly:     true,
+								Values:       nil,
+							},
+							"hs_date_entered_customer": {
+								DisplayName:  "Date entered 'Customer (Lifecycle Stage Pipeline)'",
+								ValueType:    "datetime",
+								ProviderType: "datetime.calculation_read_time",
+								ReadOnly:     true,
+								Values:       nil,
+							},
+
+							// Others.
+							"hs_notes_last_activity": {
+								DisplayName:  "Last Activity",
+								ValueType:    "other",
+								ProviderType: "object_coordinates.text",
+								ReadOnly:     true,
+								Values:       nil,
+							},
+						},
 						FieldsMap: map[string]string{
 							"address":                         "Street Address",
 							"associatedcompanyid":             "Primary Associated Company ID",
diff --git a/test/utils/mockutils/metadataResult.go b/test/utils/mockutils/metadataResult.go
index 02dd6ef1..d4e67514 100644
--- a/test/utils/mockutils/metadataResult.go
+++ b/test/utils/mockutils/metadataResult.go
@@ -3,6 +3,7 @@ package mockutils
 import (
 	"errors"
 	"fmt"
+	"reflect"
 	"strings"
 
 	"github.com/amp-labs/connectors/common"
@@ -15,7 +16,7 @@ type metadataResultComparator struct{}
 // SubsetFields checks that expected ListObjectMetadataResult fields are a subset of actual metadata result.
 func (metadataResultComparator) SubsetFields(actual, expected *common.ListObjectMetadataResult) bool {
 	if len(expected.Result) == 0 {
-		invalidTest("please specify expected FieldsMap response")
+		invalidTest("please specify expected Fields response")
 	}
 
 	for objectName, expectedMetadata := range expected.Result {
@@ -28,6 +29,18 @@ func (metadataResultComparator) SubsetFields(actual, expected *common.ListObject
 			return false
 		}
 
+		for k, v := range expectedMetadata.Fields {
+			value, ok := actualMetadata.Fields[k]
+			if !ok {
+				return false
+			}
+
+			if !reflect.DeepEqual(value, v) {
+				return false
+			}
+		}
+
+		// For backwards compatability the FieldsMap is checked alongside
 		for k, v := range expectedMetadata.FieldsMap {
 			value, ok := actualMetadata.FieldsMap[k]
 			if !ok {