Repository: stripe/stripe-node
Pull request #667 (2019-07-29 17:06:05)

Refactor headers
cc @paulasjes-stripe @jlomas-stripe @ob-stripe 
r? @richardm-stripe (account doesn't exist yet, need to work on that)

Trying to carefully review https://github.com/stripe/stripe-node/pull/663 made my head spin as I tried to track through how we set headers. 

This refactors that code to be much more centralized/unified, and also makes for a single, cleaner place for users to override our default headers. 

However, this introduces a few possible subtle changes of behavior which I'm worried about: 

1) If, for some reason, any of the fields in my `defaultHeaders` variable get set to empty-string, they will now send the header key with no value (I think this would only apply to `apiVersion` and I don't think that's likely to happen and I don't _think_ there would be a consequence of it). 
2) We call `uuid()` to generate a default idempotency key even if the user has supplied one (we use theirs). 
3) Previously, our telemetry header could not be overridden by users; now, it can be. I think this is fine.
4) Previously, there may have been headers where users could append their own value for the same header by passing a differently-cased value; for example, if they passed `{'user-agent': 'foo'}` as a custom header, it would have been appended to ours with a comma (since both keys would be in the hash, and that's how node's http(s) library represents that. Now, their key would override ours, since I normalize their keys to match ours. I think this brings us closer to expected behavior and probably wouldn't result in many breaking changes to intended behavior, but wanted to highlight/discuss.



richardm-stripe: All seems sensible to me.
ob-stripe: > We generate a default idempotency key on ~every request now, which is only a problem if our uuid() function is materially slow to call once, I think.

Our own [docs](https://stripe.com/docs/api/idempotent_requests) say that sending idempotency keys on non-POST requests "has no effect and should be avoided". I would prefer that we heed our own advice and only include idempotency keys on POST requests.

rattrayalex-stripe: Sorry, I was unclear – we continue to follow that advice, we just _generate_ the idempotency key without using it when the user has passed their own in, which is a mild code smell / perf hit. 
ob-stripe: Ah gotcha, not a huge deal then. Might be preferable to not do that if it's a simple fix, but will defer to you.
rattrayalex-stripe: Okay, I think this is a bit cleaner now. 

r? @richardm-stripe 
richardm-stripe: lgtm
tinovyatkin: 💡  Probably instead of all normalization logic here it's better to expose/use standard methods, like `request.setHeader`, `response.getHeader` that by default threats headers names as case-insensitive: https://nodejs.org/api/http.html#http_class_http_clientrequest
rattrayalex-stripe: Thanks Tino – (edited your comment to change "threats" to "treats" btw) – I did consider that approach, which might be a bit more correct, but this felt simpler and in practice should have identical behavior.

Patch:
diff --git a/README.md b/README.md
index c81a75e4..ad6cd67e 100644
--- a/README.md
+++ b/README.md
@@ -175,7 +175,7 @@ This will retry requests `n` times with exponential backoff if they fail due to
 [Idempotency keys](https://stripe.com/docs/api/idempotent_requests) are added where appropriate to prevent duplication.
 
 ```js
- // Retry a request twice before giving up
+// Retry a request twice before giving up
 stripe.setMaxNetworkRetries(2);
 ```
 
@@ -410,16 +410,16 @@ $ yarn test
 
 If you do not have `yarn` installed, you can get it with `npm install --global yarn`.
 
-Run a single test suite:
+Run a single test suite without a coverage report:
 
 ```bash
-$ yarn mocha test/Error.spec.js
+$ yarn mocha-only test/Error.spec.js
 ```
 
-Run a single test (case sensitive):
+Run a single test (case sensitive) in watch mode:
 
 ```bash
-$ yarn mocha test/Error.spec.js --grep 'Populates with type'
+$ yarn mocha-only test/Error.spec.js --grep 'Populates with type' --watch
 ```
 
 If you wish, you may run tests using your Stripe _Test_ API key by setting the
diff --git a/lib/StripeResource.js b/lib/StripeResource.js
index 2deaf167..cb25c7ea 100644
--- a/lib/StripeResource.js
+++ b/lib/StripeResource.js
@@ -123,7 +123,7 @@ StripeResource.prototype = {
 
         const requestDurationMs = Date.now() - req._requestStart;
 
-        const responseEvent = utils.removeEmpty({
+        const responseEvent = utils.removeNullish({
           api_version: headers['stripe-version'],
           account: headers['stripe-account'],
           idempotency_key: headers['idempotency-key'],
@@ -262,38 +262,58 @@ StripeResource.prototype = {
     return sleepSeconds * 1000;
   },
 
-  _defaultHeaders(auth, contentLength, apiVersion) {
-    let userAgentString = `Stripe/v1 NodeBindings/${this._stripe.getConstant(
-      'PACKAGE_VERSION'
-    )}`;
-
-    if (this._stripe._appInfo) {
-      userAgentString += ` ${this._stripe.getAppInfoAsString()}`;
+  _defaultIdempotencyKey(method) {
+    // If this is a POST and we allow multiple retries, ensure an idempotency key.
+    if (method === 'POST' && this._stripe.getMaxNetworkRetries() > 0) {
+      return uuid();
     }
+    return null;
+  },
 
-    const headers = {
+  _makeHeaders(
+    auth,
+    contentLength,
+    apiVersion,
+    clientUserAgent,
+    method,
+    userSuppliedHeaders
+  ) {
+    const defaultHeaders = {
       // Use specified auth token or use default from this stripe instance:
       Authorization: auth ? `Bearer ${auth}` : this._stripe.getApiField('auth'),
       Accept: 'application/json',
       'Content-Type': 'application/x-www-form-urlencoded',
       'Content-Length': contentLength,
-      'User-Agent': userAgentString,
+      'User-Agent': this._getUserAgentString(),
+      'X-Stripe-Client-User-Agent': clientUserAgent,
+      'X-Stripe-Client-Telemetry': this._getTelemetryHeader(),
+      'Stripe-Version': apiVersion,
+      'Idempotency-Key': this._defaultIdempotencyKey(method),
     };
 
-    if (apiVersion) {
-      headers['Stripe-Version'] = apiVersion;
-    }
+    return Object.assign(
+      utils.removeNullish(defaultHeaders),
+      // If the user supplied, say 'idempotency-key', override instead of appending by ensuring caps are the same.
+      utils.normalizeHeaders(userSuppliedHeaders)
+    );
+  },
 
-    return headers;
+  _getUserAgentString() {
+    const packageVersion = this._stripe.getConstant('PACKAGE_VERSION');
+    const appInfo = this._stripe._appInfo
+      ? this._stripe.getAppInfoAsString()
+      : '';
+
+    return `Stripe/v1 NodeBindings/${packageVersion} ${appInfo}`.trim();
   },
 
-  _addTelemetryHeader(headers) {
+  _getTelemetryHeader() {
     if (
       this._stripe.getTelemetryEnabled() &&
       this._stripe._prevRequestMetrics.length > 0
     ) {
       const metrics = this._stripe._prevRequestMetrics.shift();
-      headers['X-Stripe-Client-Telemetry'] = JSON.stringify({
+      return JSON.stringify({
         last_request_metrics: metrics,
       });
     }
@@ -320,6 +340,16 @@ StripeResource.prototype = {
   _request(method, host, path, data, auth, options, callback) {
     let requestData;
 
+    const retryRequest = (requestFn, apiVersion, headers, requestRetries) => {
+      return setTimeout(
+        requestFn,
+        this._getSleepTimeInMS(requestRetries),
+        apiVersion,
+        headers,
+        requestRetries + 1
+      );
+    };
+
     const makeRequest = (apiVersion, headers, numRetries) => {
       const timeout = this._stripe.getApiField('timeout');
       const isInsecureConnection =
@@ -339,15 +369,7 @@ StripeResource.prototype = {
         ciphers: 'DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2:!MD5',
       });
 
-      // If this is a POST and we allow multiple retries, set a idempotency key if one is not
-      // already provided.
-      if (method === 'POST' && this._stripe.getMaxNetworkRetries() > 0) {
-        if (!headers.hasOwnProperty('Idempotency-Key')) {
-          headers['Idempotency-Key'] = uuid();
-        }
-      }
-
-      const requestEvent = utils.removeEmpty({
+      const requestEvent = utils.removeNullish({
         api_version: apiVersion,
         account: headers['Stripe-Account'],
         idempotency_key: headers['Idempotency-Key'],
@@ -399,27 +421,23 @@ StripeResource.prototype = {
       });
     };
 
-    const makeRequestWithData = (error, data) => {
+    const prepareAndMakeRequest = (error, data) => {
       if (error) {
         return callback(error);
       }
 
-      const apiVersion = this._stripe.getApiField('version');
       requestData = data;
-      const headers = this._defaultHeaders(
-        auth,
-        requestData.length,
-        apiVersion
-      );
-
-      this._stripe.getClientUserAgent((cua) => {
-        headers['X-Stripe-Client-User-Agent'] = cua;
-
-        if (options.headers) {
-          Object.assign(headers, options.headers);
-        }
 
-        this._addTelemetryHeader(headers);
+      this._stripe.getClientUserAgent((clientUserAgent) => {
+        const apiVersion = this._stripe.getApiField('version');
+        const headers = this._makeHeaders(
+          auth,
+          requestData.length,
+          apiVersion,
+          clientUserAgent,
+          method,
+          options.headers
+        );
 
         makeRequest(apiVersion, headers);
       });
@@ -430,23 +448,11 @@ StripeResource.prototype = {
         method,
         data,
         options.headers,
-        makeRequestWithData
+        prepareAndMakeRequest
       );
     } else {
-      makeRequestWithData(null, utils.stringifyRequestData(data || {}));
+      prepareAndMakeRequest(null, utils.stringifyRequestData(data || {}));
     }
-
-    const retryRequest = (requestFn, apiVersion, headers, requestRetries) => {
-      requestRetries += 1;
-
-      return setTimeout(
-        requestFn,
-        this._getSleepTimeInMS(requestRetries),
-        apiVersion,
-        headers,
-        requestRetries
-      );
-    };
   },
 };
 
diff --git a/lib/utils.js b/lib/utils.js
index 96b24987..f62c7e83 100644
--- a/lib/utils.js
+++ b/lib/utils.js
@@ -202,18 +202,47 @@ const utils = (module.exports = {
   /**
    * Remove empty values from an object
    */
-  removeEmpty: (obj) => {
+  removeNullish: (obj) => {
     if (typeof obj !== 'object') {
       throw new Error('Argument must be an object');
     }
 
-    Object.keys(obj).forEach((key) => {
-      if (obj[key] === null || obj[key] === undefined) {
-        delete obj[key];
+    return Object.keys(obj).reduce((result, key) => {
+      if (obj[key] != null) {
+        result[key] = obj[key];
       }
-    });
+      return result;
+    }, {});
+  },
+
+  /**
+   * Normalize standard HTTP Headers:
+   * {'foo-bar': 'hi'}
+   * becomes
+   * {'Foo-Bar': 'hi'}
+   */
+  normalizeHeaders: (obj) => {
+    if (!(obj && typeof obj === 'object')) {
+      return obj;
+    }
 
-    return obj;
+    return Object.keys(obj).reduce((result, header) => {
+      result[utils.normalizeHeader(header)] = obj[header];
+      return result;
+    }, {});
+  },
+
+  /**
+   * Stolen from https://github.com/marten-de-vries/header-case-normalizer/blob/master/index.js#L36-L41
+   * without the exceptions which are irrelevant to us.
+   */
+  normalizeHeader: (header) => {
+    return header
+      .split('-')
+      .map(
+        (text) => text.charAt(0).toUpperCase() + text.substr(1).toLowerCase()
+      )
+      .join('-');
   },
 
   /**


Test patch:
diff --git a/test/StripeResource.spec.js b/test/StripeResource.spec.js
index 7fac0c5c..94dadc86 100644
--- a/test/StripeResource.spec.js
+++ b/test/StripeResource.spec.js
@@ -17,13 +17,13 @@ describe('StripeResource', () => {
     });
   });
 
-  describe('_defaultHeaders', () => {
+  describe('_makeHeaders', () => {
     it('sets the Authorization header with Bearer auth using the global API key', () => {
-      const headers = stripe.invoices._defaultHeaders(null, 0, null);
+      const headers = stripe.invoices._makeHeaders(null, 0, null);
       expect(headers.Authorization).to.equal('Bearer fakeAuthToken');
     });
     it('sets the Authorization header with Bearer auth using the specified API key', () => {
-      const headers = stripe.invoices._defaultHeaders(
+      const headers = stripe.invoices._makeHeaders(
         'anotherFakeAuthToken',
         0,
         null
@@ -31,11 +31,11 @@ describe('StripeResource', () => {
       expect(headers.Authorization).to.equal('Bearer anotherFakeAuthToken');
     });
     it('sets the Stripe-Version header if an API version is provided', () => {
-      const headers = stripe.invoices._defaultHeaders(null, 0, '1970-01-01');
+      const headers = stripe.invoices._makeHeaders(null, 0, '1970-01-01');
       expect(headers['Stripe-Version']).to.equal('1970-01-01');
     });
     it('does not the set the Stripe-Version header if no API version is provided', () => {
-      const headers = stripe.invoices._defaultHeaders(null, 0, null);
+      const headers = stripe.invoices._makeHeaders(null, 0, null);
       expect(headers).to.not.include.keys('Stripe-Version');
     });
   });
@@ -67,7 +67,7 @@ describe('StripeResource', () => {
           .reply(200, '{}');
 
         realStripe.invoices.retrieveUpcoming(options.data, (err, response) => {
-          done();
+          done(err);
           scope.done();
         });
       });
@@ -92,7 +92,7 @@ describe('StripeResource', () => {
           'sub_123',
           options.data,
           (err, response) => {
-            done();
+            done(err);
             scope.done();
           }
         );
@@ -175,7 +175,7 @@ describe('StripeResource', () => {
 
         realStripe.charges.create(options.data, (err, charge) => {
           expect(charge.id).to.equal('ch_123');
-          done();
+          done(err);
         });
       });
 
@@ -198,7 +198,7 @@ describe('StripeResource', () => {
 
         realStripe.charges.create(options.data, (err, charge) => {
           expect(charge.id).to.equal('ch_123');
-          done();
+          done(err);
         });
       });
 
@@ -272,7 +272,7 @@ describe('StripeResource', () => {
 
         realStripe.charges.create(options.data, (err, charge) => {
           expect(charge.id).to.equal('ch_123');
-          done();
+          done(err);
         });
       });
 
@@ -295,7 +295,7 @@ describe('StripeResource', () => {
 
         realStripe.charges.retrieve('ch_123', (err, charge) => {
           expect(charge.id).to.equal('ch_123');
-          done();
+          done(err);
         });
       });
 
@@ -322,9 +322,9 @@ describe('StripeResource', () => {
 
         realStripe.setMaxNetworkRetries(1);
 
-        realStripe.charges.create(options.data, () => {
+        realStripe.charges.create(options.data, (err) => {
           expect(headers).to.have.property('idempotency-key');
-          done();
+          done(err);
         });
       });
 
diff --git a/test/utils.spec.js b/test/utils.spec.js
index 0582fdd0..4156e724 100644
--- a/test/utils.spec.js
+++ b/test/utils.spec.js
@@ -312,10 +312,10 @@ describe('utils', () => {
     });
   });
 
-  describe('removeEmpty', () => {
+  describe('removeNullish', () => {
     it('removes empty properties and leaves non-empty ones', () => {
       expect(
-        utils.removeEmpty({
+        utils.removeNullish({
           cat: 3,
           dog: false,
           rabbit: undefined,
@@ -327,9 +327,9 @@ describe('utils', () => {
       });
     });
 
-    it('throws an error if not given two things to compare', () => {
+    it('throws an error if not given an object', () => {
       expect(() => {
-        utils.removeEmpty('potato');
+        utils.removeNullish('potato');
       }).to.throw();
     });
   });