Project: Apache Hadoop (App Definition and Development / Database)
File: connecting.md

<! Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- MACRO{toc|fromDepth=0|toDepth=2} --> This document covers how to connect to and authenticate with S3 stores, primarily AWS S3. There have been changes in this mechanism between the V1 and V2 SDK, in particular specifying the region is now preferred to specifying the regional S3 endpoint. For connecting to third-party stores, please read after reading this document. AWS provides storage, compute and other services around the world, in regions. Data in S3 is stored buckets; each bucket is a single region. There are some "special" regions: China, AWS GovCloud. It is believed that the S3A connector works in these places, at least to the extent that nobody has complained about it not working. The S3A connector connects to Amazon S3 storage over HTTPS connections, either directly or through an HTTP proxy. HTTP HEAD and GET, PUT, POST and DELETE requests are invoked to perform different read/write operations against the store. There are multiple ways to connect to an S3 bucket To an ; an HTTPS server hosted by amazon or a third party. To a FIPS-compliant S3 Endpoint. To an AWS S3 . Through a VPC connection, . AWS . The S3A connector supports all these; S3 Endpoints are the primary mechanism used -either explicitly declared or automatically determined from the declared region of the bucket. Not supported: AWS . As of December 2023, AWS S3 uses Transport Layer Security (TLS) to secure the communications channel; the S3A client is does this through the Apache . Third-party stores implementing the S3 API are also supported. These often only implement a subset of the S3 API; not all features are available. If TLS authentication is used, then the HTTPS certificates for the private stores MUST be installed on the JVMs on hosts within the Hadoop cluster. See after reading this document. There are three core settings to connect to an S3 store, endpoint, region and whether or not to use path style access. ```xml <property> <name>fs.s3a.endpoint</name> <description>AWS S3 endpoint to connect to. An up-to-date list is provided in the AWS Documentation: regions and endpoints. Without this property, the endpoint/hostname of the S3 Store is inferred from the value of fs.s3a.endpoint.region, fs.s3a.endpoint.fips and more. </description> </property> <property> <name>fs.s3a.endpoint.region</name> <value>REGION</value> <description>AWS Region of the data</description> </property> <property> <name>fs.s3a.path.style.access</name> <value>false</value> <description>Enable S3 path style access by disabling the default virtual hosting behaviour. Needed for AWS PrivateLink, S3 AccessPoints, and, generally, third party stores. Default: false. </description> </property> ``` Historically the S3A connector has preferred the endpoint as defined by the option `fs.s3a.endpoint`. With the move to the AWS V2 SDK, there is more emphasis on the region, set by the `fs.s3a.endpoint.region` option. Normally, declaring the region in `fs.s3a.endpoint.region` should be sufficient to set up the network connection to correctly connect to an AWS-hosted S3 store. Configs `fs.s3a.endpoint` and `fs.s3a.endpoint.region` are used to set values for S3 endpoint and region respectively. If `fs.s3a.endpoint.region` is configured with valid AWS region value, S3A will configure the S3 client to use this

If this is set to a region that does not match your bucket, you will receive a 301 redirect response. If `fs.s3a.endpoint.region` is not set and `fs.s3a.endpoint` is set with valid endpoint value, S3A will attempt to parse the region from the endpoint and configure S3 client to use the region value. If both `fs.s3a.endpoint` and `fs.s3a.endpoint.region` are not set, S3A will use `us-east-2` as default region and enable cross region access. In this case, S3A does not attempt to override the endpoint while configuring the S3 client. If `fs.s3a.endpoint` is not set and `fs.s3a.endpoint.region` is set to an empty string, S3A will configure S3 client without any region or endpoint override. This will allow fallback to S3 SDK region resolution chain. More details . If `fs.s3a.endpoint` is set to central endpoint `s3.amazonaws.com` and `fs.s3a.endpoint.region` is not set, S3A will use `us-east-2` as default region and enable cross region access. In this case, S3A does not attempt to override the endpoint while configuring the S3 client. If `fs.s3a.endpoint` is set to central endpoint `s3.amazonaws.com` and `fs.s3a.endpoint.region` is also set to some region, S3A will use that region value and enable cross region access. In this case, S3A does not attempt to override the endpoint while configuring the S3 client. When the cross region access is enabled while configuring the S3 client, even if the region set is incorrect, S3 SDK determines the region. This is done by making the request, and if the SDK receives 301 redirect response, it determines the region at the cost of a HEAD request, and caches it. Please note that some endpoint and region settings that require cross region access are complex and improving over time. Hence, they may be considered unstable. If you are working with third party stores, please check . See . ```xml <property> <name>fs.s3a.connection.maximum</name> <value>200</value> <description>Controls the maximum number of simultaneous connections to S3. This must be bigger than the value of fs.s3a.threads.max so as to stop threads being blocked waiting for new HTTPS connections. </description> </property> <property> <name>fs.s3a.connection.ssl.enabled</name> <value>true</value> <description> Enables or disables SSL connections to AWS services. </description> </property> <property> <name>fs.s3a.ssl.channel.mode</name> <value>Default_JSSE</value> <description> TLS implementation and cipher options. Values: OpenSSL, Default, DefaultJSSE, DefaultJSSEwithGCM Default_JSSE is not truly the the default JSSE implementation because the GCM cipher is disabled when running on Java 8. However, the name was not changed in order to preserve backwards compatibility. Instead, new mode called DefaultJSSEwith_GCM delegates to the default JSSE implementation with no changes to the list of enabled ciphers. OpenSSL requires the wildfly JAR on the classpath and a compatible installation of the openssl binaries. It is often faster than the JVM libraries, but also trickier to use. </description> </property> <property> <name>fs.s3a.socket.send.buffer</name> <value>8192</value> <description> Socket send buffer hint to amazon connector. Represented in bytes. </description> </property> <property> <name>fs.s3a.socket.recv.buffer</name> <value>8192</value> <description> Socket receive buffer hint to amazon connector. Represented in bytes. </description> </property> ``` Connections to S3A stores can be made through an HTTP or HTTPS proxy. ```xml <property> <name>fs.s3a.proxy.host</name> <description> Hostname of the (optional) proxy server for S3 connections. </description> </property> <property> <name>fs.s3a.proxy.ssl.enabled</name> <value>false</value> <description> Does the proxy use a TLS connection? </description> </property> <property> <name>fs.s3a.proxy.port</name> <description> Proxy server port. If this property is not set but fs.s3a.proxy.host is, port 80 or 443 is assumed (consistent with the value of fs.s3a.connection.ssl.enabled). </description> </property> <property> <name>fs.s3a.proxy.username</name> <description>Username for authenticating with proxy server.</description> </property> <property> <name>fs.s3a.proxy.password</name> <description>Password for authenticating with proxy server.</description> </property> <property> <name>fs.s3a.proxy.domain</name> <description>Domain for authenticating with proxy server.</description> </property> <property> <name>fs.s3a.proxy.workstation</name> <description>Workstation for authenticating with proxy server.</description> </property> ``` Sometimes the proxy can be source of problems, especially if HTTP connections are kept in the connection pool for some time. Experiment with the values of

and `fs.s3a.connection.request.timeout` if long-lived connections have problems. S3 Buckets are hosted in different "regions", the default being "US-East-1". The S3A client talks to this region by default, issuing HTTP requests to the server `s3.amazonaws.com`. S3A can work with buckets from any region. Each region has its own S3 endpoint, documented . Applications running in EC2 infrastructure do not pay for IO to/from local S3 buckets. They will be billed for access to remote buckets. Always use local buckets and local copies of data, wherever possible. With the V4 signing protocol, AWS requires the explicit region endpoint to be used hence S3A must be configured to use the specific endpoint. This is done by setting the regon in the configuration option `fs.s3a.endpoint.region`, or by explicitly setting `fs.s3a.endpoint` and `fs.s3a.endpoint.region`. All endpoints other than the default region only support interaction with buckets local to that S3 instance. Standard S3 buckets support "cross-region" access where use of the original `us-east-1` endpoint allows access to the data, but newer storage types, particularly S3 Express are not supported. If the wrong endpoint is used, the request will fail. This may be reported as a 301/redirect error, or as a 400 Bad Request: take these as cues to check the endpoint setting of a bucket. The up to date list of regions is . This list can be used to specify the endpoint of individual buckets, for example for buckets in the us-west-2 and EU/Ireland endpoints. ```xml <property> <name>fs.s3a.bucket.us-west-2-dataset.endpoint.region</name> <value>us-west-2</value> </property> <property> <name>fs.s3a.bucket.eu-dataset.endpoint.region</name> <value>eu-west-1</value> </property> ``` allows for a private connection to a bucket to be defined, with network access rules managing how a bucket can be accessed. Follow the documentation to create the private link retrieve the DNS name from the console, such as `vpce-f264a96c-6d27bfa7c85e.s3.us-west-2.vpce.amazonaws.com` Convert this to an endpoint URL by prefixing "https://bucket." Declare this as the bucket endpoint and switch to path-style access. Declare the region: there is no automated determination of the region from the `vpce` URL. ```xml <property> <name>fs.s3a.bucket.example-usw2.endpoint</name> <value>https://bucket.vpce-f264a96c-6d27bfa7c85e.s3.us-west-2.vpce.amazonaws.com/</value> </property> <property> <name>fs.s3a.bucket.example-usw2.path.style.access</name> <value>true</value> </property> <property> <name>fs.s3a.bucket.example-usw2.endpoint.region</name> <value>us-west-2</value> </property> ``` It is possible to use endpoints which support a restricted subset of TLS algorithms. Amazon provide a specific set of to use so callers can be confident that the network communication is compliant with the standard: non-compliant algorithms are unavailable. The boolean option `fs.s3a.endpoint.fips` (default `false`) switches the S3A connector to using the FIPS endpoint of a region. ```xml <property> <name>fs.s3a.endpoint.fips</name> <value>true</value> <description>Use the FIPS endpoint</description> </property> ``` For a single bucket: ```xml <property> <name>fs.s3a.bucket.noaa-isd-pds.endpoint.fips</name> <value>true</value> <description>Use the FIPS endpoint for the NOAA dataset</description> </property> ``` If `fs.s3a.endpoint.fips` is `true`, the endpoint option `fs.s3a.endpoint` MUST NOT be set to any non-central endpoint value. If `fs.s3a.endpoint.fips` is `true`, the only optionally allowed value for `fs.s3a.endpoint` is central endpoint `s3.amazonaws.com`. S3A error message if `s3.eu-west-2.amazonaws.com` endpoint is used with FIPS: ``` Non central endpoint cannot be set when fs.s3a.endpoint.fips is true : https://s3.eu-west-2.amazonaws.com ``` S3A validation is used to fail-fast before the SDK returns error. AWS SDK error message if S3A does not fail-fast: ``` A custom endpoint cannot be combined with FIPS: https://s3.eu-west-2.amazonaws.com ``` The SDK calculates the FIPS-specific endpoint without any awareness as to whether FIPs is supported by a region. The first attempt to interact with the service will fail ``` java.net.UnknownHostException: software.amazon.awssdk.core.exception.SdkClientException: Received an UnknownHostException when attempting to interact with a service. See cause for the exact endpoint that is failing to resolve. If this is happening on an endpoint that previously worked, there may be a network connectivity issue or your DNS cache could be storing endpoints for too long.:

``` For more details on endpoint and region settings, please check . Important OpenSSL and FIPS endpoints Linux distributions with an FIPS-compliant SSL library may not be compatible with wildfly. Always use with the JDK SSL implementation unless you are confident that the library is compatible, or wish to experiment with the settings outside of production deployments. ```xml <property> <name>fs.s3a.ssl.channel.mode</name> <value>Default_JSSE</value> </property> ``` S3A supports usage which improves VPC integration with S3 and simplifies your data's permission model because different policies can be applied now on the Access Point level. For more information about why to use and how to create them make sure to read the official documentation. Accessing data through an access point, is done by using its ARN, as opposed to just the bucket name. You can set the Access Point ARN property using the following per bucket configuration property: ```xml <property> <name>fs.s3a.bucket.sample-bucket.accesspoint.arn</name> <value> {ACCESSPOINTARNHERE} </value> <description>Configure S3a traffic to use this AccessPoint</description> </property> ``` This configures access to the `sample-bucket` bucket for S3A, to go through the new Access Point ARN. So, for example `s3a://sample-bucket/key` will now use your configured ARN when getting data from S3 instead of your bucket. the name of the bucket used in the s3a:// URLs is irrelevant; it is not used when connecting with the store Example ```xml <property> <name>fs.s3a.bucket.example-ap.accesspoint.arn</name> <value>arn:aws:s3:eu-west-2:152813717728:accesspoint/ap-example-london</value> <description>AccessPoint bound to bucket name example-ap</description> </property> ``` The `fs.s3a.accesspoint.required` property can also require all access to S3 to go through Access Points. This has the advantage of increasing security inside a VPN / VPC as you only allow access to known sources of data defined through Access Points. In case there is a need to access a bucket directly (without Access Points) then you can use per bucket overrides to disable this setting on a bucket by bucket basis i.e. `fs.s3a.bucket.{YOUR-BUCKET}.accesspoint.required`. ```xml <!-- Require access point only access --> <property> <name>fs.s3a.accesspoint.required</name> <value>true</value> </property> <!-- Disable it on a per-bucket basis if needed --> <property> <name>fs.s3a.bucket.example-bucket.accesspoint.required</name> <value>false</value> </property> ``` Before using Access Points make sure you're not impacted by the following: The endpoint for S3 requests will automatically change to use `s3-accesspoint.REGION.amazonaws.{com | com.cn}` depending on the Access Point ARN. While considering endpoints, if you have any custom signers that use the host endpoint property make sure to update them if needed; The `storediag` command within the utility JAR is recommended as the way to view and print settings. If `storediag` doesn't connect to your S3 store, nothing else will. Based on the experience of people who field support calls, here are some of the main connectivity issues which cause problems. All hosts in the cluster need to have the configuration secrets; local environment variables are not enough. If HTTPS/TLS is used for a private store, the relevant certificates MUST be installed everywhere. For applications such as distcp, the options need to be passed with the job. If your cluster is configured to use a private store, AWS-hosted buckets are not visible. If you wish to read access in a private store, you need to change the endpoint. Private S3 stores generally expect path style access. These usually surface rapidly and with meaningful messages. Region errors generally surface as `UnknownHostException` `AWSRedirectException` "Received permanent redirect response to region" Endpoint configuration problems can be more varied, as they are just HTTPS URLs. When it works, it is fast. But it is fussy as to openSSL implementations, TLS protocols and more. Because it uses the native openssl binaries, operating system updates can trigger regressions. Disabling it should be the first step to troubleshooting any TLS problems. If there is a proxy, set it