HDFS (Hadoop) Connector Reference | MuleSoft Documentation
# HDFS (Hadoop) Connector Reference
Anypoint Connector for Hadoop (HDFS) v6.0
Release Notes: Hadoop (HDFS) Connector Release Notes
## HDFS Configurations
### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
The name for this configuration. Connectors reference the configuration with this name.  
**Connection** | 

|  The connection types to provide to this configuration.  
**Expiration Policy** |  Expiration Policy |  Configures the minimum amount of time that a dynamic configuration instance can remain idle before Mule considers it eligible for expiration. This does not mean that the platform expires the instance at the exact moment that it becomes eligible. Mule purges the instances as appropriate.  
## Connection Types
To access the data in a Hadoop (HDFS) instance, you must authenticate your application’s requests using a supported authentication method.
For Hadoop (HDFS) Connector, the following connection authentication types are supported:


### Kerberos
Use the following fields in the Global Element Properties dialog window to configure the **Kerberos** connection type:
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
The Kerberos principal. The Username is passed to the HDFS client as the `hadoop.job.ugi` configuration entry. The Username can be overridden by values you specify in **Configuration Resources** and **Configuration Entries**. This parameter is called Username for backward compatibility reasons.  
**Keytab Path** |  Enter the path to the keytab file associated with the Username you specified. The path is used to obtain a ticket granting ticket (TGT) from the authorization server. If this value is not provided Kerberos looks for a TGT associated with the specified Username within your local Kerberos cache.  
**Name Node Uri** |  The name of the file system to connect to. The Name Node is passed to the HDFS client as the `FileSystem#FS_DEFAULT_NAME_KEY` configuration entry. The Name Node can be overridden by values you specify in **Configuration Resources** and **Configuration Entries**.  
**Configuration Resources** |  Array of String |  A `java.util.List` of configuration resource files for the HDFS client to load. You can provide additional configuration files, for example, `core-site.xml`.  
**Configuration Entries** |  A `java.util.Map` of configuration entries to use by the HDFS client. You can provide additional configuration entries as key-value pairs.  
**Reconnection** |  |  When the application is deployed, a connectivity test is performed on all connectors. If reconnection is enabled, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy.  
### Simple
Use the following fields in the Global Element Properties dialog window to configure the **Simple** connection type:
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
User identity that Hadoop uses for permissions in HDFS. When Simple authentication is used, Hadoop requires that the user is set as a system property called `HADOOP_USER_NAME`. If a value is not provided, Hadoop uses the username of the OS user who is currently logged in.  
**Name Node Uri** |  The name of the file system to which to connect. The Name Node is passed to the HDFS client as the `FileSystem#FS_DEFAULT_NAME_KEY` configuration entry. The Name Node can be overridden by values you specify in **Configuration Resources** and **Configuration Entries**.  
**Configuration Resources** |  Array of String |  A `java.util.List` of configuration resource files for the HDFS client to load. You can provide additional configuration files, for example, `core-site.xml`.  
**Configuration Entries** |  A `java.util.Map` of configuration entries for the HDFS client to use. You can provide additional configuration entries as key-value pairs.  
**Reconnection** |  |  When the application is deployed, a connectivity test is performed on all connectors. If reconnection is enabled, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy.  
## Operations
  * Copy From Local File
  * Copy To Local File
  * Delete Directory
  *   *   *   *   * Make Directories
  * Read Operation
  *   * Set Permission


### Associated Sources
### Append
`<hdfs:append>`
Append the current payload to a file located at the designated path. **Note:** by default the Hadoop server has the append option disabled. To append data to an existing file, refer to the dfs.support.append configuration parameter.
#### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
**Configuration** |  The name of the configuration to use.  
The path of the file to write to.  
**Buffer Size** |  The buffer size to use when appending to the file.  
The payload to append to the file.  
**Reconnection Strategy** | 
  *   * Reconnect Forever

|  A retry strategy in case of connectivity errors  
### For Configurations
HDFS Configurations
#### Throws
  * HDFS:CONNECTIVITY
  * HDFS:INVALID_REQUEST_DATA
  * HDFS:INVALID_STRUCTURE_FOR_INPUT_DATA
  * HDFS:RETRY_EXHAUSTED
  * HDFS:UNKNOWN


### Copy From Local File
`<hdfs:copy-from-local-file>`
Copy the source file on the local disk to the file system for a target path, set **Delete Source** if the source file should be removed.
#### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
**Configuration** |  The name of the configuration to use.  
**Delete Source** |  Whether to delete the source.  
Whether to overwrite destination content.  
The source path in the file system.  
**Destination** |  The target path on the local disk.  
**Reconnection Strategy** | 
  *   * Reconnect Forever

|  A retry strategy in case of connectivity errors  
### For Configurations
HDFS Configurations
#### Throws
  * HDFS:CONNECTIVITY
  * HDFS:INVALID_REQUEST_DATA
  * HDFS:INVALID_STRUCTURE_FOR_INPUT_DATA
  * HDFS:RETRY_EXHAUSTED
  * HDFS:UNKNOWN


### Copy To Local File
`<hdfs:copy-to-local-file>`
Copy the source file in the file system to a local disk at the given target path. Set **Delete Source** if the source file should be removed. **Use Raw Local File System** indicates whether to use RawLocalFileSystem as it is a non-CRC file system.
#### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
**Configuration** |  The name of the configuration to use.  
**Delete Source** |  Whether to delete the source.  
**Use Raw Local File System** |  Whether to use RawLocalFileSystem as a local file system.  
The source path on the File System.  
**Destination** |  The target path on the local disk.  
**Reconnection Strategy** | 
  *   * Reconnect Forever

|  A retry strategy in case of connectivity errors  
### For Configurations
HDFS Configurations
#### Throws
  * HDFS:CONNECTIVITY
  * HDFS:INVALID_REQUEST_DATA
  * HDFS:INVALID_STRUCTURE_FOR_INPUT_DATA
  * HDFS:RETRY_EXHAUSTED
  * HDFS:UNKNOWN


### Delete Directory
`<hdfs:delete-directory>`
Delete the file or directory located at the designated path.
#### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
**Configuration** |  The name of the configuration to use.  
The path of the file to delete.  
**Reconnection Strategy** | 
  *   * Reconnect Forever

|  A retry strategy in case of connectivity errors  
### For Configurations
HDFS Configurations
#### Throws
  * HDFS:CONNECTIVITY
  * HDFS:INVALID_REQUEST_DATA
  * HDFS:INVALID_STRUCTURE_FOR_INPUT_DATA
  * HDFS:RETRY_EXHAUSTED
  * HDFS:UNKNOWN


### Delete File
`<hdfs:delete-file>`
Delete the file or directory located at the designated path.
#### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
**Configuration** |  The name of the configuration to use.  
The path of the file to delete.  
**Reconnection Strategy** | 
  *   * Reconnect Forever

|  A retry strategy in case of connectivity errors  
### For Configurations
HDFS Configurations
#### Throws
  * HDFS:CONNECTIVITY
  * HDFS:INVALID_REQUEST_DATA
  * HDFS:INVALID_STRUCTURE_FOR_INPUT_DATA
  * HDFS:RETRY_EXHAUSTED
  * HDFS:UNKNOWN


### Get Metadata
`<hdfs:get-metadata>`
Get the metadata of a path
#### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
**Configuration** |  The name of the configuration to use.  
The path of the file to delete.  
**Target Variable** |  The name of a variable to store the operation’s output.  
**Target Value** |  An expression to evaluate against the operation’s output and store the expression outcome in the target variable.  
**Reconnection Strategy** | 
  *   * Reconnect Forever
### Delete File
`<hdfs:delete-file>`
Delete the file or directory located at the designated path.
#### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
**Configuration** |  The name of the configuration to use.  
The path of the file to delete.  
**Reconnection Strategy** | 
  *   * Reconnect Forever

|  A retry strategy in case of connectivity errors  
### For Configurations
HDFS Configurations
#### Throws
  * HDFS:CONNECTIVITY
  * HDFS:INVALID_REQUEST_DATA
  * HDFS:INVALID_STRUCTURE_FOR_INPUT_DATA
  * HDFS:RETRY_EXHAUSTED
  * HDFS:UNKNOWN


### Get Metadata
`<hdfs:get-metadata>`
Get the metadata of a path
#### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
**Configuration** |  The name of the configuration to use.  
The path of the file to delete.  
**Target Variable** |  The name of a variable to store the operation’s output.  
**Target Value** |  An expression to evaluate against the operation’s output and store the expression outcome in the target variable.  
**Reconnection Strategy** | 
  *   * Reconnect Forever

|  A retry strategy in case of connectivity errors  
#### Output
### For Configurations
HDFS Configurations
#### Throws
  * HDFS:CONNECTIVITY
  * HDFS:INVALID_REQUEST_DATA
  * HDFS:INVALID_STRUCTURE_FOR_INPUT_DATA
  * HDFS:RETRY_EXHAUSTED
  * HDFS:UNKNOWN


### Glob Status
`<hdfs:glob-status>`
Return all the files that match file pattern and are not checksum files. Results are sorted by their names.
#### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
**Configuration** |  The name of the configuration to use.  
**Path Pattern** |  A regular expression specifying the path pattern.  
The user supplied path filter  
**Target Variable** |  The name of a variable to store the operation’s output.  
**Target Value** |  An expression to evaluate against the operation’s output and store the expression outcome in the target variable.  
**Reconnection Strategy** | 
  *   * Reconnect Forever

|  A retry strategy in case of connectivity errors  
#### Output
Array of File Status  
---  
### For Configurations
HDFS Configurations
#### Throws
  * HDFS:CONNECTIVITY
  * HDFS:RETRY_EXHAUSTED


### List Status
`<hdfs:list-status>`
List the statuses of the files and directories in the given path if the path is a directory.
#### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
**Configuration** |  The name of the configuration to use.  
The given path  
The user supplied path filter  
**Target Variable** |  The name of a variable to store the operation’s output.  
**Target Value** |  An expression to evaluate against the operation’s output and store the expression outcome in the target variable.  
**Reconnection Strategy** | 
  *   * Reconnect Forever

|  A retry strategy in case of connectivity errors  
#### Output
Array of File Status  
---  
### For Configurations
HDFS Configurations
#### Throws
  * HDFS:CONNECTIVITY
  * HDFS:INVALID_REQUEST_DATA
  * HDFS:INVALID_STRUCTURE_FOR_INPUT_DATA
  * HDFS:RETRY_EXHAUSTED
  * HDFS:UNKNOWN


### Make Directories
`<hdfs:make-directories>`
Make the given file and all non-existent parents into directories. Has the semantics of Unix 'mkdir -p'. Existence of the directory hierarchy is not an error.
#### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
**Configuration** |  The name of the configuration to use.  
The path to create one or more directories.  
**Permission** |  The file system permission to use when creating the directories, either in octal or symbolic format (umask).  
**Reconnection Strategy** | 
  *   * Reconnect Forever

|  A retry strategy in case of connectivity errors.  
### For Configurations
HDFS Configurations
#### Throws
  * HDFS:CONNECTIVITY
  * HDFS:INVALID_REQUEST_DATA
  * HDFS:INVALID_STRUCTURE_FOR_INPUT_DATA
  * HDFS:RETRY_EXHAUSTED
  * HDFS:UNKNOWN


### Read Operation
`<hdfs:read-operation>`
Read the content of a file designated by its path and streams it to the rest of the flow.
#### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
**Configuration** |  The name of the configuration to use.  
The path of the file to read.  
**Buffer Size** |  The buffer size to use when reading the file.  
**Streaming Strategy** | 
  * Repeatable In Memory Stream
  * Repeatable File Store Stream
  * non-repeatable-stream

|  Configure if repeatable streams should be used and their behavior  
**Target Variable** |  The name of a variable to store the operation’s output.  
**Target Value** |  An expression to evaluate against the operation’s output and store the expression outcome in the target variable  
**Reconnection Strategy** | 
  *   * Reconnect Forever

|  A retry strategy in case of connectivity errors.  
#### Output
### For Configurations
HDFS Configurations
#### Throws
  * HDFS:CONNECTIVITY
  * HDFS:INVALID_REQUEST_DATA
  * HDFS:INVALID_STRUCTURE_FOR_INPUT_DATA
  * HDFS:RETRY_EXHAUSTED
  * HDFS:UNKNOWN


### Rename
`<hdfs:rename>`
Renames path target to path destination.
#### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
**Configuration** |  The name of the configuration to use.  
The source path to be renamed.  
**Destination** |  New path after rename.  
**Reconnection Strategy** | 
  *   * Reconnect Forever

|  A retry strategy in case of connectivity errors.  
### For Configurations
HDFS Configurations
#### Throws
  * HDFS:CONNECTIVITY
  * HDFS:INVALID_REQUEST_DATA
  * HDFS:INVALID_STRUCTURE_FOR_INPUT_DATA
  * HDFS:RETRY_EXHAUSTED
  * HDFS:UNKNOWN


### Set Owner
`<hdfs:set-owner>`
Set owner of a path for a file or a directory. The Ownername and Groupname cannot both be null.
#### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
**Configuration** |  The name of the configuration to use.  
The path of the file or directory to set owner.  
If it is null, the original username remains unchanged.  
If it is null, the original groupname remains unchanged.  
**Reconnection Strategy** | 
  *   * Reconnect Forever

|  A retry strategy in case of connectivity errors.  
### For Configurations
HDFS Configurations
#### Throws
  * HDFS:CONNECTIVITY
  * HDFS:INVALID_REQUEST_DATA
  * HDFS:INVALID_STRUCTURE_FOR_INPUT_DATA
  * HDFS:RETRY_EXHAUSTED
  * HDFS:UNKNOWN


### Set Permission
`<hdfs:set-permission>`
Set permission of a path, that is, for a file or a directory.
#### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
**Configuration** |  The name of the configuration to use.  
The path of the file or directory to set permission.  
**Permission** |  The file system permission to be set.  
**Reconnection Strategy** | 
  *   * Reconnect Forever

|  A retry strategy in case of connectivity errors.  
### For Configurations
HDFS Configurations
#### Throws
  * HDFS:CONNECTIVITY
  * HDFS:INVALID_REQUEST_DATA
  * HDFS:INVALID_STRUCTURE_FOR_INPUT_DATA
  * HDFS:RETRY_EXHAUSTED
  * HDFS:UNKNOWN


### Write
`<hdfs:write>`
Write the current payload to the designated path, either creating a new file or appending to an existing one.
#### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
**Configuration** |  The name of the configuration to use.  
the payload to write to the file.  
The path of the file to write to.  
**Permission** |  The file system permission to use if a new file is created, either in octal or symbolic format (umask).  
If a pre-existing file should be overwritten with the new content.  
**Buffer Size** |  The buffer size to use when appending to the file.  
**Replication** |  Block replication for the file.  
**Block Size** |  The block size to use when appending to the file.  
**Owner User Name** |  The username owner of the file.  
**Owner Group Name** |  The group owner of the file.  
**Reconnection Strategy** | 
  *   * Reconnect Forever
|  A retry strategy in case of connectivity errors.  
### For Configurations
HDFS Configurations
#### Throws
  * HDFS:CONNECTIVITY
  * HDFS:INVALID_REQUEST_DATA
  * HDFS:INVALID_STRUCTURE_FOR_INPUT_DATA
  * HDFS:RETRY_EXHAUSTED
  * HDFS:UNKNOWN


### Write
`<hdfs:write>`
Write the current payload to the designated path, either creating a new file or appending to an existing one.
#### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
**Configuration** |  The name of the configuration to use.  
the payload to write to the file.  
The path of the file to write to.  
**Permission** |  The file system permission to use if a new file is created, either in octal or symbolic format (umask).  
If a pre-existing file should be overwritten with the new content.  
**Buffer Size** |  The buffer size to use when appending to the file.  
**Replication** |  Block replication for the file.  
**Block Size** |  The block size to use when appending to the file.  
**Owner User Name** |  The username owner of the file.  
**Owner Group Name** |  The group owner of the file.  
**Reconnection Strategy** | 
  *   * Reconnect Forever

|  A retry strategy in case of connectivity errors.  
### For Configurations
HDFS Configurations
#### Throws
  * HDFS:CONNECTIVITY
  * HDFS:INVALID_REQUEST_DATA
  * HDFS:INVALID_STRUCTURE_FOR_INPUT_DATA
  * HDFS:RETRY_EXHAUSTED
  * HDFS:UNKNOWN


## Sources
### Read
`<hdfs:read>`
#### Parameters
Name | Type | Description | Default Value | Required  
---|---|---|---|---  
**Configuration** |  The name of the configuration to use.  
Read the content of a file designated by its path  
**Buffer Size**  
**Primary Node Only** |  Whether this source should be executed only on the primary node when running in a cluster.  
**Streaming Strategy** | 
  * Repeatable In Memory Stream
  * Repeatable File Store Stream
  * non-repeatable-stream

|  Configure if repeatable streams should be used and their behavior  
**Redelivery Policy** |  Redelivery Policy |  Defines a policy for processing the redelivery of the same message.  
**Reconnection Strategy** | 
  *   * Reconnect Forever

|  A retry strategy in case of connectivity errors.  
#### Output
Attributes Type  
---  
### For Configurations
HDFS Configurations
## Types
### Reconnection
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
**Fails Deployment** |  When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment fails if the test doesn’t pass after exhausting the associated reconnection strategy.  
**Reconnection Strategy** | 
  *   * Reconnect Forever

|  The reconnection strategy to use.  
### Reconnect
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
How often to reconnect (in milliseconds)  
The number of reconnection attempts to make  
If false, the reconnection strategy runs in a separate, non-blocking thread.  
### Reconnect Forever
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
How often in milliseconds to reconnect  
If false, the reconnection strategy runs in a separate, non-blocking thread.  
### Expiration Policy
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
**Max Idle Time** |  A scalar time value for the maximum amount of time a dynamic configuration instance should be allowed to be idle before it’s considered eligible for expiration.  
Enumeration, one of:
  * NANOSECONDS
  * MICROSECONDS
  * MILLISECONDS
  * SECONDS
  * MINUTES
  * HOURS

|  A time unit that qualifies the maxIdleTime attribute  
### Repeatable In Memory Stream
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
**Initial Buffer Size** |  The amount of memory that will be allocated to consume the stream and provide random access to it. If the stream contains more data than can be fit into this buffer, then the buffer expands according to the bufferSizeIncrement attribute, with an upper limit of maxInMemorySize.  
**Buffer Size Increment** |  This is by how much the buffer size expands if it exceeds its initial size. Setting a value of zero or lower means that the buffer should not expand, meaning that a STREAM_MAXIMUM_SIZE_EXCEEDED error is raised when the buffer gets full.  
**Max Buffer Size** |  The maximum amount of memory to use. If more than that is used, the STREAM_MAXIMUM_SIZE_EXCEEDED error is raised. A value lower than or equal to zero means no limit.  
**Buffer Unit** |  Enumeration, one of: |  The unit in which all these attributes are expressed  
### Repeatable File Store Stream
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
**Max In Memory Size** |  Defines the maximum memory that the stream should use to keep data in memory. If more memory is consumed, content on the disk is buffered.  
**Buffer Unit** |  Enumeration, one of: |  The unit in which maxInMemorySize is expressed.  
### Redelivery Policy
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
**Max Redelivery Count** |  The maximum number of times a message can be redelivered and processed unsuccessfully before triggering a process-failed message.  
**Use Secure Hash** |  Whether to use a secure hash algorithm to identify a redelivered message.  
**Message Digest Algorithm** |  The secure hashing algorithm to use. If not set, the default is SHA-256.  
**Id Expression** |  Defines one or more expressions to use to determine when a message has been redelivered. This property can only be set if **Use Secure Hash** is false.  
**Object Store** |  Object Store |  The object store where the redelivery counter for each message is stored.  
### Metadata
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
**Check Summary** |   
**Content Summary** |  Content Summary  
**File Status**  
**Path Exists**  
### Check Summary
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
**Bytes Per CRC**  
**Crc Per Block**  
### Content Summary
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
**Directory Count**  
**Snapshot Directory Count**  
**Snapshot File Count**  
**Snapshot Length**  
**Snapshot Space Consumed**  
### File Status
Field | Type | Description | Default Value | Required  
---|---|---|---|---  
**Access Time** |  Access time of file in milliseconds  
**Block Replication** |  Replication factor of file  
**Block Size** |  Block size of file  
Indicates whether path is a directory  
Group owner associated with file  
Length of file in bytes  
**Modification Time** |  Modification time of file in milliseconds  
Owner of file  
**Permission** |  Permission of file as an octal string  
**Symbolic Link** |  Indicates whether a path is a symbolic link  
## See Also
MuleSoft Help Center