Repository: aws/aws-sdk-js
Pull request #3269 (2020-05-21 22:29:10)

fix issues with required endpoint discovery feature
Previously, SDK will throw errors if `endpointDiscoveryEnabled` 
is not `true` when calling operations that **require** endpoint 
discovery. Setting the config to `false`(default value) doesn't mean
anything to SDK behavior.
This PR changes the behavior in following ways:

* For a given service; if at least one operation requires endpoint
  discovery then the SDK enables endpoint discovery by default
  for all operations of that service, including operations where
  endpoint discovery is optional.

* Users can set config `endpointDiscoveryEnabled`, env `AWS_ENDPOINT_DISCOVERY_ENABLED`,
  config file entry `endpoint_discovery_enabled` to `false` to
  explictly disable endpoint discovery, even operations the require
  endpoint discovery will fail;

* SDK throws if endpoint discovery is explicitly disabled but
  operation requires endpoint discovery;

This PR also include another minor fix:

* Now SDK throws more clear error message when required endpoint
  operation fails

<!--
Thank you for your pull request. Please provide a description below.
-->

##### Checklist
<!-- Remove items that do not apply. For completed items, change [ ] to [x]. -->

- [x] `npm run test` passes
- [x] changelog is added, `npm run add-change`
- [x] run `bundle exec rake docs:api` and inspect `doc/latest/index.html` if documentation is changed


aws-sdk-js-automation: 
### AWS CodeBuild CI Report

* CodeBuild project: sdk-v2-github
* Commit ID: 31c572284a8080e1616a6eca61143fa301182915
* Result: FAILED
* [Build Logs](https://r6vdd33du7.execute-api.us-west-2.amazonaws.com/Prod/buildlogs?key=9162ba38-2f08-408b-9587-ef3eaf60aedb%2Fbuild.log) (available for 30 days)

*Powered by [github-codebuild-logs](https://serverlessrepo.aws.amazon.com/applications/arn:aws:serverlessrepo:us-east-1:277187709615:applications~github-codebuild-logs), available on the [AWS Serverless Application Repository](https://aws.amazon.com/serverless/serverlessrepo/)*


<!--
CREATED BY GITHUB-CODEBUILD-LOGS
-->


aws-sdk-js-automation: 
### AWS CodeBuild CI Report

* CodeBuild project: sdk-v2-github
* Commit ID: 2cabc9e513397f8dd769cd671de91c5acb9a8758
* Result: FAILED
* [Build Logs](https://r6vdd33du7.execute-api.us-west-2.amazonaws.com/Prod/buildlogs?key=67e2d036-ff59-4a98-93d7-6a471d47e234%2Fbuild.log) (available for 30 days)

*Powered by [github-codebuild-logs](https://serverlessrepo.aws.amazon.com/applications/arn:aws:serverlessrepo:us-east-1:277187709615:applications~github-codebuild-logs), available on the [AWS Serverless Application Repository](https://aws.amazon.com/serverless/serverlessrepo/)*


<!--
CREATED BY GITHUB-CODEBUILD-LOGS
-->


aws-sdk-js-automation: 
### AWS CodeBuild CI Report

* CodeBuild project: sdk-v2-github
* Commit ID: 08c17cf550f155454b5e680ef0ca683f8c0bf593
* Result: SUCCEEDED
* [Build Logs](https://r6vdd33du7.execute-api.us-west-2.amazonaws.com/Prod/buildlogs?key=781b75f1-4d25-4596-ad7c-fcb03c55388b%2Fbuild.log) (available for 30 days)

*Powered by [github-codebuild-logs](https://serverlessrepo.aws.amazon.com/applications/arn:aws:serverlessrepo:us-east-1:277187709615:applications~github-codebuild-logs), available on the [AWS Serverless Application Repository](https://aws.amazon.com/serverless/serverlessrepo/)*


<!--
CREATED BY GITHUB-CODEBUILD-LOGS
-->


aws-sdk-js-automation: 
### AWS CodeBuild CI Report

* CodeBuild project: sdk-v2-github
* Commit ID: b5259ac41d3f9097ffdc7c177a1392fe3db36ca0
* Result: SUCCEEDED
* [Build Logs](https://r6vdd33du7.execute-api.us-west-2.amazonaws.com/Prod/buildlogs?key=fe761af5-0d8b-4941-92ba-683659eae9a9%2Fbuild.log) (available for 30 days)

*Powered by [github-codebuild-logs](https://serverlessrepo.aws.amazon.com/applications/arn:aws:serverlessrepo:us-east-1:277187709615:applications~github-codebuild-logs), available on the [AWS Serverless Application Repository](https://aws.amazon.com/serverless/serverlessrepo/)*


<!--
CREATED BY GITHUB-CODEBUILD-LOGS
-->


aws-sdk-js-automation: 
### AWS CodeBuild CI Report

* CodeBuild project: sdk-v2-github
* Commit ID: b18bc626d32d95841f668154bf33778967e08f75
* Result: SUCCEEDED
* [Build Logs](https://r6vdd33du7.execute-api.us-west-2.amazonaws.com/Prod/buildlogs?key=63d1e066-4551-43ab-a7b7-c43e7a47714f%2Fbuild.log) (available for 30 days)

*Powered by [github-codebuild-logs](https://serverlessrepo.aws.amazon.com/applications/arn:aws:serverlessrepo:us-east-1:277187709615:applications~github-codebuild-logs), available on the [AWS Serverless Application Repository](https://aws.amazon.com/serverless/serverlessrepo/)*


<!--
CREATED BY GITHUB-CODEBUILD-LOGS
-->



Patch:
diff --git a/.changes/next-release/bugfix-DualStack-a5f57529.json b/.changes/next-release/bugfix-DualStack-a5f57529.json
new file mode 100644
index 000000000..104874391
--- /dev/null
+++ b/.changes/next-release/bugfix-DualStack-a5f57529.json
@@ -0,0 +1,5 @@
+{
+  "type": "bugfix",
+  "category": "DualStack",
+  "description": "Add dualstack by modifying existing endpoint in config"
+}
\ No newline at end of file
diff --git a/.changes/next-release/bugfix-EndpointDiscovery-f55d6800.json b/.changes/next-release/bugfix-EndpointDiscovery-f55d6800.json
new file mode 100644
index 000000000..e34a2ef1b
--- /dev/null
+++ b/.changes/next-release/bugfix-EndpointDiscovery-f55d6800.json
@@ -0,0 +1,5 @@
+{
+  "type": "bugfix",
+  "category": "EndpointDiscovery",
+  "description": "* If at least one operation requires endpoint discovery then the SDK enables endpoint discovery by default including operational endpoint discovery operations; * Users can set config `endpointDiscoveryEnabled`, env `AWS_ENDPOINT_DISCOVERY_ENABLED`, config file entry `endpoint_discovery_enabled` to `false` to explictly disable endpoint discovery, even operations the require endpoint discovery will fail; * SDK throws if endpoint discovery is explicitly disabled but operation requires endpoint discovery; * Now SDK throws more clear error message when required endpoint operation fails"
+}
\ No newline at end of file
diff --git a/.changes/next-release/bugfix-S3-6167b5a8.json b/.changes/next-release/bugfix-S3-6167b5a8.json
new file mode 100644
index 000000000..f2970e2e2
--- /dev/null
+++ b/.changes/next-release/bugfix-S3-6167b5a8.json
@@ -0,0 +1,5 @@
+{
+  "type": "bugfix",
+  "category": "S3",
+  "description": "throw exception when incomplete body is returned from CopyObject, UploadPartCopy, and CompleteMultipartUpload"
+}
\ No newline at end of file
diff --git a/lib/config.js b/lib/config.js
index 7da15580c..a9f735226 100644
--- a/lib/config.js
+++ b/lib/config.js
@@ -164,9 +164,13 @@ var PromisesDependency;
  *     Defaults to `true`.
  *
  * @!attribute endpointDiscoveryEnabled
- *   @return [Boolean] whether to enable endpoint discovery for operations that
- *     allow optionally using an endpoint returned by the service.
- *     Defaults to 'false'
+ *   @return [Boolean|undefined] whether to call operations with endpoints
+ *     given by service dynamically. Setting this config to `true` will enable
+ *     endpoint discovery for all applicable operations. Setting it to `false`
+ *     will explicitly disable endpoint discovery even though operations that
+ *     require endpoint discovery will presumably fail. Leaving it to
+ *     `undefined` means SDK only do endpoint discovery when it's required.
+ *     Defaults to `undefined`
  *
  * @!attribute endpointCacheSize
  *   @return [Number] the size of the global cache storing endpoints from endpoint
@@ -319,10 +323,13 @@ AWS.Config = AWS.util.inherit({
    *   S3 Transfer Acceleration endpoint with the S3 service. Default: `false`.
    * @option options clientSideMonitoring [Boolean] whether to collect and
    *   publish this client's performance metrics of all its API requests.
-   * @option options endpointDiscoveryEnabled [Boolean] whether to enable endpoint
-   *   discovery for operations that allow optionally using an endpoint returned by
-   *   the service.
-   *   Defaults to 'false'
+   * @option options endpointDiscoveryEnabled [Boolean|undefined] whether to
+   *   call operations with endpoints given by service dynamically. Setting this
+   * config to `true` will enable endpoint discovery for all applicable operations.
+   *   Setting it to `false` will explicitly disable endpoint discovery even though
+   *   operations that require endpoint discovery will presumably fail. Leaving it
+   *   to `undefined` means SDK will only do endpoint discovery when it's required.
+   *   Defaults to `undefined`
    * @option options endpointCacheSize [Number] the size of the global cache storing
    *   endpoints from endpoint discovery operations. Once endpoint cache is created,
    *   updating this setting cannot change existing cache size.
@@ -553,7 +560,7 @@ AWS.Config = AWS.util.inherit({
     retryDelayOptions: {},
     useAccelerateEndpoint: false,
     clientSideMonitoring: false,
-    endpointDiscoveryEnabled: false,
+    endpointDiscoveryEnabled: undefined,
     endpointCacheSize: 1000,
     hostPrefixEnabled: true,
     stsRegionalEndpoints: 'legacy'
diff --git a/lib/discover_endpoint.js b/lib/discover_endpoint.js
index 5fe5b48ec..a29392569 100644
--- a/lib/discover_endpoint.js
+++ b/lib/discover_endpoint.js
@@ -167,19 +167,14 @@ function requiredDiscoverEndpoint(request, done) {
     }]);
     endpointRequest.send(function(err, data) {
       if (err) {
-        var errorParams = {
-          code: 'EndpointDiscoveryException',
-          message: 'Request cannot be fulfilled without specifying an endpoint',
-          retryable: false
-        };
-        request.response.error = util.error(err, errorParams);
+        request.response.error = util.error(err, { retryable: false });
         AWS.endpointCache.remove(cacheKey);
 
         //fail all the pending requests in batch
         if (requestQueue[cacheKeyStr]) {
           var pendingRequests = requestQueue[cacheKeyStr];
           util.arrayEach(pendingRequests, function(requestContext) {
-            requestContext.request.response.error = util.error(err, errorParams);
+            requestContext.request.response.error = util.error(err, { retryable: false });
             requestContext.callback();
           });
           delete requestQueue[cacheKeyStr];
@@ -264,23 +259,28 @@ function isFalsy(value) {
 }
 
 /**
- * If endpoint discovery should perform for this request when endpoint discovery is optional.
+ * If endpoint discovery should perform for this request when no operation requires endpoint
+ * discovery for the given service.
  * SDK performs config resolution in order like below:
- * 1. If turned on client configuration(default to off) then turn on endpoint discovery.
- * 2. If turned on in env AWS_ENABLE_ENDPOINT_DISCOVERY then turn on endpoint discovery.
- * 3. If turned on in shared ini config file with key 'endpoint_discovery_enabled', then
- *   turn on endpoint discovery.
+ * 1. If set in client configuration.
+ * 2. If set in env AWS_ENABLE_ENDPOINT_DISCOVERY.
+ * 3. If set in shared ini config file with key 'endpoint_discovery_enabled'.
  * @param [object] request request object.
+ * @returns [boolean|undefined] if endpoint discovery config is not set in any source, this
+ *  function returns undefined
  * @api private
  */
-function isEndpointDiscoveryEnabled(request) {
+function resolveEndpointDiscoveryConfig(request) {
   var service = request.service || {};
-  if (service.config.endpointDiscoveryEnabled === true) return true;
+  if (service.config.endpointDiscoveryEnabled !== undefined) {
+    return service.config.endpointDiscoveryEnabled;
+  }
 
   //shared ini file is only available in Node
   //not to check env in browser
-  if (util.isBrowser()) return false;
+  if (util.isBrowser()) return undefined;
 
+  // If any of recognized endpoint discovery config env is set
   for (var i = 0; i < endpointDiscoveryEnabledEnvs.length; i++) {
     var env = endpointDiscoveryEnabledEnvs[i];
     if (Object.prototype.hasOwnProperty.call(process.env, env)) {
@@ -290,7 +290,7 @@ function isEndpointDiscoveryEnabled(request) {
           message: 'environmental variable ' + env + ' cannot be set to nothing'
         });
       }
-      if (!isFalsy(process.env[env])) return true;
+      return !isFalsy(process.env[env]);
     }
   }
 
@@ -311,9 +311,9 @@ function isEndpointDiscoveryEnabled(request) {
         message: 'config file entry \'endpoint_discovery_enabled\' cannot be set to nothing'
       });
     }
-    if (!isFalsy(sharedFileConfig.endpoint_discovery_enabled)) return true;
+    return !isFalsy(sharedFileConfig.endpoint_discovery_enabled);
   }
-  return false;
+  return undefined;
 }
 
 /**
@@ -328,27 +328,35 @@ function discoverEndpoint(request, done) {
   var operations = service.api.operations || {};
   var operationModel = operations[request.operation];
   var isEndpointDiscoveryRequired = operationModel ? operationModel.endpointDiscoveryRequired : 'NULL';
-  var isEnabled = isEndpointDiscoveryEnabled(request);
-
-  if (!isEnabled) {
-    // Unless endpoint discovery is required, SDK will fallback to normal regional endpoints.
-    if (isEndpointDiscoveryRequired === 'REQUIRED') {
-      throw util.error(new Error(), {
-        code: 'ConfigurationException',
-        message: 'Endpoint Discovery is not enabled but this operation requires it.'
-      });
-    }
-    return done();
+  var isEnabled = resolveEndpointDiscoveryConfig(request);
+  var hasRequiredEndpointDiscovery = service.api.hasRequiredEndpointDiscovery;
+  if (isEnabled || hasRequiredEndpointDiscovery) {
+    // Once a customer enables endpoint discovery, the SDK should start appending
+    // the string endpoint-discovery to the user-agent on all requests.
+    request.httpRequest.appendToUserAgent('endpoint-discovery');
   }
-
-  request.httpRequest.appendToUserAgent('endpoint-discovery');
   switch (isEndpointDiscoveryRequired) {
     case 'OPTIONAL':
-      optionalDiscoverEndpoint(request);
-      request.addNamedListener('INVALIDATE_CACHED_ENDPOINTS', 'extractError', invalidateCachedEndpoints);
+      if (isEnabled || hasRequiredEndpointDiscovery) {
+        // For a given service; if at least one operation requires endpoint discovery then the SDK must enable endpoint discovery
+        // by default for all operations of that service, including operations where endpoint discovery is optional.
+        optionalDiscoverEndpoint(request);
+        request.addNamedListener('INVALIDATE_CACHED_ENDPOINTS', 'extractError', invalidateCachedEndpoints);
+      }
       done();
       break;
     case 'REQUIRED':
+      if (isEnabled === false) {
+        // For a given operation; if endpoint discovery is required and it has been disabled on the SDK client,
+        // then the SDK must return a clear and actionable exception.
+        request.response.error = util.error(new Error(), {
+          code: 'ConfigurationException',
+          message: 'Endpoint Discovery is disabled but ' + service.api.className + '.' + request.operation +
+                    '() requires it. Please check your configurations.'
+        });
+        done();
+        break;
+      }
       request.addNamedListener('INVALIDATE_CACHED_ENDPOINTS', 'extractError', invalidateCachedEndpoints);
       requiredDiscoverEndpoint(request, done);
       break;
diff --git a/lib/model/api.js b/lib/model/api.js
index a7e6330f5..d21ba4bbd 100644
--- a/lib/model/api.js
+++ b/lib/model/api.js
@@ -51,6 +51,13 @@ function Api(api, options) {
     if (operation.endpointoperation === true) {
       property(self, 'endpointOperation', util.string.lowerFirst(name));
     }
+    if (operation.endpointdiscovery && !self.hasRequiredEndpointDiscovery) {
+      property(
+        self,
+        'hasRequiredEndpointDiscovery',
+        operation.endpointdiscovery.required === true
+      );
+    }
   }
 
   property(this, 'operations', new Collection(api.operations, options, function(name, operation) {
diff --git a/lib/region_config.js b/lib/region_config.js
index 79052c1a9..d6f189fc6 100644
--- a/lib/region_config.js
+++ b/lib/region_config.js
@@ -50,7 +50,10 @@ function configureEndpoint(service) {
       // set dualstack endpoint
       if (service.config.useDualstack && util.isDualstackAvailable(service)) {
         config = util.copy(config);
-        config.endpoint = '{service}.dualstack.{region}.amazonaws.com';
+        config.endpoint = config.endpoint.replace(
+          /{service}\.({region}\.)?/,
+          '{service}.dualstack.{region}.'
+        );
       }
 
       // set global endpoint
diff --git a/lib/services/s3.js b/lib/services/s3.js
index 87ba9a8de..b98c6997a 100644
--- a/lib/services/s3.js
+++ b/lib/services/s3.js
@@ -652,19 +652,7 @@ AWS.util.update(AWS.S3.prototype, {
   extractErrorFrom200Response: function extractErrorFrom200Response(resp) {
     if (!operationsWith200StatusCodeError[resp.request.operation]) return;
     var httpResponse = resp.httpResponse;
-    if (
-      resp.request.operation === 'copyObject' &&
-      (httpResponse.body === undefined || httpResponse.body.length === 0)
-    ) {
-      // When body is empty, S3 might stop the request on detecting client
-      // abort the request.
-      // TODO: Whether need to apply this to all the operationsWith200StatusCodeError
-      resp.data = null;
-      throw AWS.util.error(new Error(), {
-        code: 'InternalError',
-        message: 'S3 aborted request'
-      });
-    } else if (httpResponse.body && httpResponse.body.toString().match('<Error>')) {
+    if (httpResponse.body && httpResponse.body.toString().match('<Error>')) {
       // Response body with '<Error>...</Error>' indicates an exception.
       // Get S3 client object. In ManagedUpload, this.service refers to
       // S3 client object.
@@ -672,6 +660,14 @@ AWS.util.update(AWS.S3.prototype, {
       var service = this.service ? this.service : this;
       service.extractError(resp);
       throw resp.error;
+    } else if (!httpResponse.body || !httpResponse.body.toString().match(/<[\w_]/)) {
+      // When body is empty or incomplete, S3 might stop the request on detecting client
+      // side aborting the request.
+      resp.data = null;
+      throw AWS.util.error(new Error(), {
+        code: 'InternalError',
+        message: 'S3 aborted request'
+      });
     }
   },
 
diff --git a/scripts/region-checker/whitelist.js b/scripts/region-checker/whitelist.js
index 6bd3f98a4..8b1a05b90 100644
--- a/scripts/region-checker/whitelist.js
+++ b/scripts/region-checker/whitelist.js
@@ -4,9 +4,9 @@ var whitelist = {
         25,
         85,
         86,
-        197,
-        251,
-        252
+        201,
+        255,
+        256
     ],
     '/credentials/cognito_identity_credentials.js': [
         78,
@@ -35,13 +35,13 @@ var whitelist = {
         358,
         371,
         377,
-        765,
-        767,
-        886,
-        897,
-        898,
-        899,
-        904
+        761,
+        763,
+        882,
+        893,
+        894,
+        895,
+        900
     ]
 };
 


Test patch:
diff --git a/test/discover_endpoint.spec.js b/test/discover_endpoint.spec.js
index fb7a381a8..f03a1ac26 100644
--- a/test/discover_endpoint.spec.js
+++ b/test/discover_endpoint.spec.js
@@ -7,12 +7,12 @@ var iniLoader = AWS.util.iniLoader;
 var getCacheKey = endpoint_discovery_module.getCacheKey;
 var marshallCustomIdentifiers = endpoint_discovery_module.marshallCustomIdentifiers;
 
-
 var api = {
   metadata: {
     apiVersion: '2018-09-19',
     endpointPrefix: 'mockservice',
     serviceId: 'MockService',
+    serviceAbbreviation: 'MockService',
     protocol: 'json'
   },
   operations: {
@@ -118,6 +118,23 @@ var api = {
     }
   }
 };
+// API with only optional endpoint discovery operations
+var optionalEDOnlyApi = {
+  metadata: api.metadata,
+  operations: {
+    DescribeEndpoints: api.operations.DescribeEndpoints,
+    ComplexEDOperation: api.operations.ComplexEDOperation,
+    OptionalEDOperation: api.operations.OptionalEDOperation
+  }
+};
+// API with only required endpoint discovery operations
+var requiredEDOnlyApi = {
+  metadata: api.metadata,
+  operations: {
+    DescribeEndpoints: api.operations.DescribeEndpoints,
+    RequiredEDOperation: api.operations.RequiredEDOperation
+  }
+};
 
 describe('endpoint discovery', function() {
   afterEach(function() {
@@ -178,7 +195,7 @@ describe('endpoint discovery', function() {
     });
   });
 
-  describe('optionalDiscoverEndpoint', function() {
+  describe('optionalEndpointDiscovery', function() {
     beforeEach(function() {
       AWS.endpointCache.empty();
     });
@@ -192,7 +209,7 @@ describe('endpoint discovery', function() {
       var client = new AWS.Service({
         endpointDiscoveryEnabled: true,
         region: 'fake-region-1',
-        apiConfig: new AWS.Model.Api(api)
+        apiConfig: new AWS.Model.Api(optionalEDOnlyApi)
       });
       helpers.mockHttpResponse(200, {}, '{}');
       var request = client.makeRequest('optionalEDOperation', {Query: 'query'});
@@ -210,7 +227,7 @@ describe('endpoint discovery', function() {
       var client = new AWS.Service({
         endpointDiscoveryEnabled: true,
         region: 'fake-region-1',
-        apiConfig: new AWS.Model.Api(api)
+        apiConfig: new AWS.Model.Api(optionalEDOnlyApi)
       });
       var request = client.makeRequest('optionalEDOperation', {Query: 'query'});
       expect(request.httpRequest.endpoint.hostname).to.eql('mockservice.fake-region-1.amazonaws.com');
@@ -221,7 +238,7 @@ describe('endpoint discovery', function() {
         endpointDiscoveryEnabled: true,
         region: 'fake-region-1',
         maxRetries: 0,
-        apiConfig: new AWS.Model.Api(api)
+        apiConfig: new AWS.Model.Api(optionalEDOnlyApi)
       });
       var spy = helpers.spyOn(AWS.endpointCache, 'put').andCallThrough();
       var request = client.makeRequest('optionalEDOperation', {Query: 'query'});
@@ -261,7 +278,7 @@ describe('endpoint discovery', function() {
         endpointDiscoveryEnabled: true,
         region: 'fake-region-1',
         maxRetries: 0,
-        apiConfig: new AWS.Model.Api(api)
+        apiConfig: new AWS.Model.Api(optionalEDOnlyApi)
       });
       client.makeRequest('optionalEDOperation', {Query: 'query'}).send();
       client.makeRequest('optionalEDOperation', {Query: 'query'}).send();
@@ -273,7 +290,7 @@ describe('endpoint discovery', function() {
       var client = new AWS.Service({
         endpointDiscoveryEnabled: true,
         maxRetries: 0,
-        apiConfig: new AWS.Model.Api(api)
+        apiConfig: new AWS.Model.Api(optionalEDOnlyApi)
       });
       helpers.mockHttpResponse(400, {}, '');
       var spy = helpers.spyOn(AWS.endpointCache, 'put').andCallThrough();
@@ -294,7 +311,7 @@ describe('endpoint discovery', function() {
       var client = new AWS.Service({
         region: 'fake-region-1',
         endpointDiscoveryEnabled: true,
-        apiConfig: new AWS.Model.Api(api)
+        apiConfig: new AWS.Model.Api(optionalEDOnlyApi)
       });
       helpers.mockHttpResponse(400, {}, '{"__type":"InvalidEndpointException", "Message":"Error Message" }');
       var cacheKey = {
@@ -316,7 +333,7 @@ describe('endpoint discovery', function() {
     it('add "x-amz-api-version" header to discovery endpoint operation in optional endpoint discovery', function() {
       var client = new AWS.Service({
         endpointDiscoveryEnabled: true,
-        apiConfig: new AWS.Model.Api(api),
+        apiConfig: new AWS.Model.Api(optionalEDOnlyApi),
       });
       var https = require('https');
       helpers.spyOn(https, 'request').andCallThrough();
@@ -334,31 +351,30 @@ describe('endpoint discovery', function() {
     });
   });
 
-  describe('requiredDiscoveryEndpoint', function() {
+  describe('requiredEndpointDiscovery', function() {
     beforeEach(function() {
       AWS.endpointCache.empty();
     });
 
     it('fail when endpoint discovery fails', function() {
       var client = new AWS.Service({
-        endpointDiscoveryEnabled: true,
         maxRetries: 0,
-        apiConfig: new AWS.Model.Api(api)
+        apiConfig: new AWS.Model.Api(requiredEDOnlyApi)
       });
-      helpers.mockHttpResponse(400, {}, '');
+      var body = '{"message": "failed to discover endpoint", "__type": "EndpointNotReady"}';
+      helpers.mockHttpResponse(400, {}, body);
       var request = client.makeRequest('requiredEDOperation', {Query: 'query', Record: 'record'});
       request.send(function(err, data) {
         expect(err).not.to.eql(null);
         expect(data).to.eql(null);
-        expect(err.code).to.eql('EndpointDiscoveryException');
-        expect(err.message).to.eql('Request cannot be fulfilled without specifying an endpoint');
+        expect(err.code).to.eql('EndpointNotReady');
+        expect(err.message).to.eql('failed to discover endpoint');
       });
     });
 
     it('endpoint operation will be called at the same rate of api requests', function() {
       var client = new AWS.Service({
-        endpointDiscoveryEnabled: true,
-        apiConfig: new AWS.Model.Api(api)
+        apiConfig: new AWS.Model.Api(requiredEDOnlyApi)
       });
       helpers.mockHttpResponse(400, {}, '');
       var makeRequestSpy = helpers.spyOn(client, 'makeRequest').andCallThrough();
@@ -380,8 +396,7 @@ describe('endpoint discovery', function() {
 
     it('put in endpoint cache and use returned endpoint when endpoint discovery succeeds', function() {
       var client = new AWS.Service({
-        endpointDiscoveryEnabled: true,
-        apiConfig: new AWS.Model.Api(api)
+        apiConfig: new AWS.Model.Api(requiredEDOnlyApi)
       });
       var request = client.makeRequest('requiredEDOperation', {Query: 'query', Record: 'record'});
       expect(request.httpRequest.endpoint.hostname).to.eql('mockservice.mock-region.amazonaws.com');
@@ -413,8 +428,7 @@ describe('endpoint discovery', function() {
 
     it('evict invalid endpoint records from cache if InvalidEndpointException encountered', function() {
       var client = new AWS.Service({
-        endpointDiscoveryEnabled: true,
-        apiConfig: new AWS.Model.Api(api)
+        apiConfig: new AWS.Model.Api(requiredEDOnlyApi)
       });
       helpers.mockHttpResponse(400, {}, '{"__type":"InvalidEndpointException", "Message":"Error Message" }');
       var cacheKey = {
@@ -436,8 +450,7 @@ describe('endpoint discovery', function() {
 
     it('add "x-amz-api-version" header to discovery endpoint operation in required endpoint discovery', function() {
       var client = new AWS.Service({
-        endpointDiscoveryEnabled: true,
-        apiConfig: new AWS.Model.Api(api),
+        apiConfig: new AWS.Model.Api(requiredEDOnlyApi),
       });
       var https = require('https');
       helpers.spyOn(https, 'request').andCallThrough();
@@ -456,8 +469,7 @@ describe('endpoint discovery', function() {
 
     it('update "Host" header with newly discovered endpoint', function() {
       var client = new AWS.Service({
-        endpointDiscoveryEnabled: true,
-        apiConfig: new AWS.Model.Api(api),
+        apiConfig: new AWS.Model.Api(requiredEDOnlyApi),
       });
       var https = require('https');
       helpers.spyOn(https, 'request').andCallThrough();
@@ -494,8 +506,7 @@ describe('endpoint discovery', function() {
           });
 
           client = new AWS.Service({
-            endpointDiscoveryEnabled: true,
-            apiConfig: new AWS.Model.Api(api),
+            apiConfig: new AWS.Model.Api(requiredEDOnlyApi),
           });
         });
 
@@ -564,7 +575,7 @@ describe('endpoint discovery', function() {
                 endpointOperationCount++;
                 timer = setTimeout(function() {
                   res.statusCode = 400;
-                  res.write('{"__type": "endpoint is currently unavailable"}');
+                  res.write('{"__type": "EndpointUnavailable", "message": "endpoint is currently unavailable"}');
                   res.end();
                 }, 50);
               } else {
@@ -587,8 +598,8 @@ describe('endpoint discovery', function() {
             request.send(function(err, data) {
               requestsCounter ++;
               if (err) failedRequestCounter ++;
-              expect(err.code).to.eql('EndpointDiscoveryException');
-              expect(err.message).to.eql('Request cannot be fulfilled without specifying an endpoint');
+              expect(err.code).to.eql('EndpointUnavailable');
+              expect(err.message).to.eql('endpoint is currently unavailable');
               if (requestsCounter === concurrency) {
                 AWS.events.removeAllListeners();
                 server.close();
@@ -639,14 +650,14 @@ describe('endpoint discovery', function() {
 
     it('default not run endpoint operation for optional endpoint discovery', function() {
       var client = new AWS.Service({
-        apiConfig: new AWS.Model.Api(api)
+        apiConfig: new AWS.Model.Api(optionalEDOnlyApi)
       });
       var putInCacheSpy = helpers.spyOn(AWS.endpointCache, 'put').andCallThrough();
       client.makeRequest('optionalEDOperation', {Query: 'query'}).send();
       expect(putInCacheSpy.calls.length).to.eql(0);
       client = new AWS.Service({
         endpointDiscoveryEnabled: true,
-        apiConfig: new AWS.Model.Api(api)
+        apiConfig: new AWS.Model.Api(optionalEDOnlyApi)
       });
       client.makeRequest('optionalEDOperation', {Query: 'query'}).send();
       expect(putInCacheSpy.calls.length).to.eql(1);
@@ -708,14 +719,73 @@ describe('endpoint discovery', function() {
       process.env = {};
       if (AWS.util.isBrowser()) process.env = null;
       AWS.endpointCache.empty();
+      delete AWS.config.endpointDiscoveryEnabled;
     });
     afterEach(function() {
       process.env = env;
     });
 
-    it('default to turnoff endpoint discovery', function() {
+    it('will enable endpoint discovery if at least one operation requires endpoint discovery', function() {
+      var client = new AWS.Service({
+        maxRetries: 0,
+        apiConfig: new AWS.Model.Api(api)
+      });
+      var spy = helpers.createSpy('send inject');
+      AWS.events.on('sign', function(req) {
+        if (req.operation === 'describeEndpoints') {
+          spy(req);
+        }
+      });
+      var body = '{"Endpoints": [{"Address": "https://cell1.fakeservice.amazonaws.com/fakeregion", "CachePeriodInMinutes": 1}]}';
+      helpers.mockHttpResponse(200, {}, body);
+      client.makeRequest('requiredEDOperation', {Query: 'query', Record: 'record'}).send();
+      expect(spy.calls.length).to.eql(1);
+      client.makeRequest('optionalEDOperation', {Query: 'query', Record: 'record'}).send();
+      // endpoint discovery is enabled even including operations where endpoint discovery is optional.
+      expect(spy.calls.length).to.eql(2);
+    });
+
+    it('throws when endpoint discovery is required for operation but explicitly disabled from client', function() {
+      var client = new AWS.Service({
+        endpointDiscoveryEnabled: false,
+        maxRetries: 0,
+        apiConfig: new AWS.Model.Api(requiredEDOnlyApi)
+      });
+      var body = '{"Endpoints": [{"Address": "https://cell1.fakeservice.amazonaws.com/fakeregion", "CachePeriodInMinutes": 1}]}';
+      helpers.mockHttpResponse(200, {}, body);
+      var request = client.makeRequest('requiredEDOperation', {Query: 'query', Record: 'record'});
+      request.send(function(err, data) {
+        expect(err).not.to.eql(null);
+        expect(data).to.eql(null);
+        expect(err.code).to.eql('ConfigurationException');
+        expect(err.message).to.eql(
+          'Endpoint Discovery is disabled but MockService.requiredEDOperation() requires it. Please check your configurations.'
+        );
+      });
+    });
+
+    it('throws when endpoint discovery is required for operation but explicitly disabled from global config', function() {
+      AWS.config.endpointDiscoveryEnabled = false;
+      var client = new AWS.Service({
+        maxRetries: 0,
+        apiConfig: new AWS.Model.Api(requiredEDOnlyApi)
+      });
+      var body = '{"Endpoints": [{"Address": "https://cell1.fakeservice.amazonaws.com/fakeregion", "CachePeriodInMinutes": 1}]}';
+      helpers.mockHttpResponse(200, {}, body);
+      var request = client.makeRequest('requiredEDOperation', {Query: 'query', Record: 'record'});
+      request.send(function(err, data) {
+        expect(err).not.to.eql(null);
+        // expect(data).to.eql(null);
+        expect(err.code).to.eql('ConfigurationException');
+        expect(err.message).to.eql(
+          'Endpoint Discovery is disabled but MockService.requiredEDOperation() requires it. Please check your configurations.'
+        );
+      });
+    });
+
+    it('default to turnoff endpoint discovery behavior for operational endpoint discovery', function() {
       client = new AWS.Service({
-        apiConfig: new AWS.Model.Api(api),
+        apiConfig: new AWS.Model.Api(optionalEDOnlyApi),
       });
       var spy = helpers.createSpy('send inject');
       AWS.events.on('sign', function(req) {
@@ -726,18 +796,6 @@ describe('endpoint discovery', function() {
       helpers.mockHttpResponse(200, {}, '{"Endpoints": [{"Address": "https://cell1.fakeservice.amazonaws.com/fakeregion", "CachePeriodInMinutes": 1}]}');
       client.makeRequest('optionalEDOperation', {Query: 'query'}).send();
       expect(spy.calls.length).to.eql(0);
-      // Throw if operation requires endpoint discovery but the feature is disabled.
-      var request = client.makeRequest('requiredEDOperation',  {Query: 'query', Record: 'record'});
-      var error;
-      try {
-        request.send();
-      } catch (e) {
-        error = e;
-      }
-      expect(spy.calls.length).to.eql(0);
-      expect(error).not.to.eql(undefined);
-      expect(error.name).to.eql('ConfigurationException');
-      expect(error.message).to.eql('Endpoint Discovery is not enabled but this operation requires it.');
     });
 
     it('turn on endpoint discovery in client config', function() {
@@ -779,8 +837,7 @@ describe('endpoint discovery', function() {
     if (AWS.util.isNode()) {
       it('turn on endpoint discovery from environmental variable AWS_ENABLE_ENDPOINT_DISCOVERY', function() {
         var client = new AWS.Service({
-          endpointDiscoveryEnabled: false,
-          apiConfig: new AWS.Model.Api(api),
+          apiConfig: new AWS.Model.Api(optionalEDOnlyApi),
         });
         var spy = helpers.createSpy('send inject');
         AWS.events.on('sign', function(req) {
@@ -792,8 +849,6 @@ describe('endpoint discovery', function() {
         helpers.mockHttpResponse(200, {}, '{"Endpoints": [{"Address": "https://cell1.fakeservice.amazonaws.com/fakeregion", "CachePeriodInMinutes": 1}]}');
         client.makeRequest('optionalEDOperation', {Query: 'query'}).send();
         expect(spy.calls.length).to.eql(1);
-        client.makeRequest('requiredEDOperation',  {Query: 'query', Record: 'record'}).send();
-        expect(spy.calls.length).to.eql(2);
 
         process.env.AWS_ENABLE_ENDPOINT_DISCOVERY = undefined;
         request = client.makeRequest('optionalEDOperation', {Query: 'query'});
@@ -808,10 +863,28 @@ describe('endpoint discovery', function() {
         expect(error.message).to.eql('environmental variable AWS_ENABLE_ENDPOINT_DISCOVERY cannot be set to nothing');
       });
 
+      it('throws when endpoint discovery is required for operation but explicitly disabled from environmental variable', function() {
+        var client = new AWS.Service({
+          maxRetries: 0,
+          apiConfig: new AWS.Model.Api(requiredEDOnlyApi)
+        });
+        var body = '{"Endpoints": [{"Address": "https://cell1.fakeservice.amazonaws.com/fakeregion", "CachePeriodInMinutes": 1}]}';
+        helpers.mockHttpResponse(200, {}, body);
+        process.env.AWS_ENABLE_ENDPOINT_DISCOVERY = '0';
+        var request = client.makeRequest('requiredEDOperation', {Query: 'query', Record: 'record'});
+        request.send(function(err, data) {
+          expect(err).not.to.eql(null);
+          // expect(data).to.eql(null);
+          expect(err.code).to.eql('ConfigurationException');
+          expect(err.message).to.eql(
+            'Endpoint Discovery is disabled but MockService.requiredEDOperation() requires it. Please check your configurations.'
+          );
+        });
+      });
+
       it('turn on endpoint discovery from environmental variable AWS_ENDPOINT_DISCOVERY_ENABLED', function() {
         var client = new AWS.Service({
-          endpointDiscoveryEnabled: false,
-          apiConfig: new AWS.Model.Api(api),
+          apiConfig: new AWS.Model.Api(optionalEDOnlyApi),
         });
         var spy = helpers.createSpy('send inject');
         AWS.events.on('sign', function(req) {
@@ -823,8 +896,6 @@ describe('endpoint discovery', function() {
         helpers.mockHttpResponse(200, {}, '{"Endpoints": [{"Address": "https://cell1.fakeservice.amazonaws.com/fakeregion", "CachePeriodInMinutes": 1}]}');
         client.makeRequest('optionalEDOperation', {Query: 'query'}).send();
         expect(spy.calls.length).to.eql(1);
-        client.makeRequest('requiredEDOperation',  {Query: 'query', Record: 'record'}).send();
-        expect(spy.calls.length).to.eql(2);
 
         process.env.AWS_ENDPOINT_DISCOVERY_ENABLED = undefined;
         request = client.makeRequest('optionalEDOperation', {Query: 'query'});
@@ -841,8 +912,7 @@ describe('endpoint discovery', function() {
 
       it('turn on endpoint discovery from config file', function() {
         var client = new AWS.Service({
-          endpointDiscoveryEnabled: false,
-          apiConfig: new AWS.Model.Api(api),
+          apiConfig: new AWS.Model.Api(optionalEDOnlyApi),
         });
         helpers.spyOn(iniLoader, 'loadFrom').andReturn({'dummyRole': {endpoint_discovery_enabled: true}});
         process.env.AWS_PROFILE = 'dummyRole';
@@ -855,11 +925,9 @@ describe('endpoint discovery', function() {
         helpers.mockHttpResponse(200, {}, '{"Endpoints": [{"Address": "https://cell1.fakeservice.amazonaws.com/fakeregion", "CachePeriodInMinutes": 1}]}');
         client.makeRequest('optionalEDOperation', {Query: 'query'}).send();
         expect(spy.calls.length).to.eql(1);
-        client.makeRequest('requiredEDOperation',  {Query: 'query', Record: 'record'}).send();
-        expect(spy.calls.length).to.eql(2);
 
         helpers.spyOn(iniLoader, 'loadFrom').andReturn({'dummyRole': {endpoint_discovery_enabled: undefined}});
-        var request = client.makeRequest('requiredEDOperation', {Query: 'query', Record: 'record'});
+        var request = client.makeRequest('optionalEDOperation', {Query: 'query', Record: 'record'});
         var error;
         try {
           request.send();
diff --git a/test/model/api.spec.js b/test/model/api.spec.js
index 61c5ad164..982ec3c77 100644
--- a/test/model/api.spec.js
+++ b/test/model/api.spec.js
@@ -86,7 +86,7 @@
     });
 
     describe('endpoint operation', function() {
-      it('adds endpiont discovery opeation name when exists', function() {
+      it('adds endpiont discovery opeation name and required flag when exists', function() {
         var api = make({
           operations: {
             DescribeEndpoints: {
@@ -117,6 +117,7 @@
           }
         });
         expect(api.endpointOperation).to.equal('describeEndpoints');
+        expect(api.hasRequiredEndpointDiscovery).to.equal(true);
         expect(api.operations.someOperation.endpointDiscoveryRequired).to.equal('NULL');
         expect(api.operations.optionalEndpointOperation.endpointDiscoveryRequired).to.equal('OPTIONAL');
         expect(api.operations.requiredEndpointOperation.endpointDiscoveryRequired).to.equal('REQUIRED');
diff --git a/test/s3/managed_upload.spec.js b/test/s3/managed_upload.spec.js
index 58ed91390..e92349165 100644
--- a/test/s3/managed_upload.spec.js
+++ b/test/s3/managed_upload.spec.js
@@ -45,6 +45,9 @@
       AWS.S3.ManagedUpload.prototype.minPartSize = 10;
       ref1 = [], err = ref1[0], data = ref1[1];
       helpers.spyOn(AWS.S3.prototype, 'extractError').andReturn(function() {});
+      //Because we cannot mock individual body for a series of responses, so we will just disable
+      //this function. Otherwise it will make each 'completeMultipartUpload' throws because of empty body.
+      helpers.spyOn(AWS.S3.prototype, 'extractErrorFrom200Response').andReturn(function() {});
       return upload = null;
     });
     afterEach(function() {
diff --git a/test/services/s3.spec.js b/test/services/s3.spec.js
index 54b9415bd..f1ef2937a 100644
--- a/test/services/s3.spec.js
+++ b/test/services/s3.spec.js
@@ -111,6 +111,24 @@ describe('AWS.S3', function() {
         useDualstack: true
       });
       expect(s3.endpoint.hostname).to.equal('s3.dualstack.us-east-1.amazonaws.com');
+
+      s3 = new AWS.S3({
+        region: 'cn-north-1',
+        useDualstack: true
+      });
+      expect(s3.endpoint.hostname).to.equal('s3.dualstack.cn-north-1.amazonaws.com.cn');
+
+      s3 = new AWS.S3({
+        region: 'us-iso-east-1',
+        useDualstack: true
+      });
+      expect(s3.endpoint.hostname).to.equal('s3.dualstack.us-iso-east-1.c2s.ic.gov');
+
+      s3 = new AWS.S3({
+        region: 'us-isob-east-1',
+        useDualstack: true
+      });
+      expect(s3.endpoint.hostname).to.equal('s3.dualstack.us-isob-east-1.sc2s.sgov.gov');
     });
   });
 
@@ -2174,6 +2192,32 @@ describe('AWS.S3', function() {
         done();
       });
     });
+
+    it('returns an error when the resp is 200 with incomplete body', function(done) {
+      helpers.mockHttpResponse(200, {}, '<?xml version="1.0" encoding="UTF-8"?>');
+      s3.completeMultipartUpload(function(error, data) {
+        expect(error).to.be.instanceOf(Error);
+        expect(error.code).to.equal('InternalError');
+        expect(error.message).to.equal('S3 aborted request');
+        expect(error.statusCode).to.equal(200);
+        expect(error.retryable).to.equal(true);
+        expect(data).to.equal(null);
+        done();
+      });
+    });
+
+    it('returns an error when the resp is 200 with empty body', function(done) {
+      helpers.mockHttpResponse(200, {}, '');
+      s3.completeMultipartUpload(function(error, data) {
+        expect(error).to.be.instanceOf(Error);
+        expect(error.code).to.equal('InternalError');
+        expect(error.message).to.equal('S3 aborted request');
+        expect(error.statusCode).to.equal(200);
+        expect(error.retryable).to.equal(true);
+        expect(data).to.equal(null);
+        done();
+      });
+    });
   });
 
   describe('copyObject', function() {
@@ -2210,6 +2254,19 @@ describe('AWS.S3', function() {
       });
     });
 
+    it('returns an error when the resp is 200 with incomplete body', function(done) {
+      helpers.mockHttpResponse(200, {}, '<?xml version="1.0" encoding="UTF-8"?>');
+      s3.copyObject(function(error, data) {
+        expect(error).to.be.instanceOf(Error);
+        expect(error.code).to.equal('InternalError');
+        expect(error.message).to.equal('S3 aborted request');
+        expect(error.statusCode).to.equal(200);
+        expect(error.retryable).to.equal(true);
+        expect(data).to.equal(null);
+        done();
+      });
+    });
+
     it('returns an error when the resp is 200 with empty body', function(done) {
       helpers.mockHttpResponse(200, {}, '');
       s3.copyObject(function(error, data) {
@@ -2261,6 +2318,32 @@ describe('AWS.S3', function() {
         done();
       });
     });
+
+    it('returns an error when the resp is 200 with incomplete body', function(done) {
+      helpers.mockHttpResponse(200, {}, '<?xml version="1.0" encoding="UTF-8"?>');
+      s3.uploadPartCopy(function(error, data) {
+        expect(error).to.be.instanceOf(Error);
+        expect(error.code).to.equal('InternalError');
+        expect(error.message).to.equal('S3 aborted request');
+        expect(error.statusCode).to.equal(200);
+        expect(error.retryable).to.equal(true);
+        expect(data).to.equal(null);
+        done();
+      });
+    });
+
+    it('returns an error when the resp is 200 with empty body', function(done) {
+      helpers.mockHttpResponse(200, {}, '');
+      s3.uploadPartCopy(function(error, data) {
+        expect(error).to.be.instanceOf(Error);
+        expect(error.code).to.equal('InternalError');
+        expect(error.message).to.equal('S3 aborted request');
+        expect(error.statusCode).to.equal(200);
+        expect(error.retryable).to.equal(true);
+        expect(data).to.equal(null);
+        done();
+      });
+    });
   });
 
   describe('getBucketLocation', function() {
diff --git a/test/services/s3control.spec.js b/test/services/s3control.spec.js
index 74bb608fd..8e4a549e0 100644
--- a/test/services/s3control.spec.js
+++ b/test/services/s3control.spec.js
@@ -28,6 +28,13 @@ describe('AWS.S3Control', function() {
     request.send();
     expect(request.httpRequest.headers.Host).to.eql('222.s3-control.dualstack.us-east-1.amazonaws.com');
     expect(request.httpRequest.endpoint.hostname).to.eql('222.s3-control.dualstack.us-east-1.amazonaws.com');
+
+    client = new AWS.S3Control({region: 'cn-north-1', useDualstack: true});
+    request = client.getPublicAccessBlock({AccountId: '222'});
+    helpers.mockResponse({data: {}});
+    request.send();
+    expect(request.httpRequest.headers.Host).to.eql('222.s3-control.dualstack.cn-north-1.amazonaws.com.cn');
+    expect(request.httpRequest.endpoint.hostname).to.eql('222.s3-control.dualstack.cn-north-1.amazonaws.com.cn');
   });
 
   it('append accountId to hostname when supplied and using customized endpoint', function() {