Project: Beam (App Definition and Development / Streaming & Messaging)
File: cdap.md

title: "Cdap IO" <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> A `CdapIO` is a transform for reading data from source or writing data to sink CDAP plugin. `CdapIO` currently supports the following CDAP Batch plugins by referencing `CDAP plugin` class name: Also, any other CDAP Batch plugin based on Hadoop's `InputFormat` or `OutputFormat` can be used. They can be easily added to the list of supported by class name plugins, for more details please see . `CdapIO` currently supports CDAP Streaming plugins based on . Requirements for CDAP Streaming plugins: CDAP Streaming plugin should be based on `Spark Receiver` (Spark 2.4). CDAP Streaming plugin should support work with offsets. Corresponding Spark Receiver should implement interface. Records should have the numeric field that represents record offset. In order to read from CDAP plugin you will need to pass: `Key` and `Value` classes. You will need to check if these classes have a Beam Coder available. `PluginConfig` object with parameters for certain CDAP plugin. You can easily build `PluginConfig` object using `ConfigWrapper` class by specifying: Class of the needed `PluginConfig`. `Map<String, Object>` parameters map for corresponding CDAP plugin. For example: {{< highlight java >}} Map<String, Object> myPluginConfigParams = new HashMap<>(); // Read plugin parameters (e.g. from PipelineOptions) and put them into 'myPluginConfigParams' map. myPluginConfigParams.put(MyPluginConstants.USERNAMEPARAMETERNAME, pipelineOptions.getUsername()); // ... MyPluginConfig pluginConfig = new ConfigWrapper<>(MyPluginConfig.class).withParams(myPluginConfigParams).build(); {{< /highlight >}} Some CDAP plugins are already supported and can be used just by plugin class name. For example: {{< highlight java >}} CdapIO.Read<NullWritable, JsonElement> readTransform = CdapIO.<NullWritable, JsonElement>read() .withCdapPluginClass(HubspotBatchSource.class) .withPluginConfig(pluginConfig) .withKeyClass(NullWritable.class) .withValueClass(JsonElement.class); p.apply("read", readTransform); {{< /highlight >}} If CDAP plugin is not supported by plugin class name, you can easily build `Plugin` object by passing the following parameters: Class of CDAP Batch plugin. The `InputFormat` class used to connect to your CDAP plugin of choice. The `InputFormatProvider` class used to provide `InputFormat`. Then you will be able to pass this `Plugin` object to `CdapIO`. For example: {{< highlight java >}} CdapIO.Read<String, String> readTransform = CdapIO.<String, String>read() .withCdapPlugin( Plugin.createBatch( MyCdapPlugin.class, MyInputFormat.class, MyInputFormatProvider.class)) .withPluginConfig(pluginConfig) .withKeyClass(String.class) .withValueClass(String.class); p.apply("read", readTransform); {{< /highlight >}} {{< highlight java >}} SourceHubspotConfig pluginConfig = new ConfigWrapper<>(SourceHubspotConfig.class).withParams(pluginConfigParams).build(); CdapIO<NullWritable, JsonElement> readTransform = CdapIO.<NullWritable, JsonElement>read() .withCdapPluginClass(HubspotBatchSource.class) .withPluginConfig(pluginConfig) .withKeyClass(NullWritable.class) .withValueClass(JsonElement.class); p.apply("readFromHubspotPlugin", readTransform); {{< /highlight >}} {{< highlight java >}} SalesforceSourceConfig pluginConfig = new ConfigWrapper<>(SalesforceSourceConfig.class).withParams(pluginConfigParams).build(); CdapIO<Schema, LinkedHashMap> readTransform = CdapIO.<Schema, LinkedHashMap>read() .withCdapPluginClass(SalesforceBatchSource.class) .withPluginConfig(pluginConfig) .withKeyClass(Schema.class) .withValueClass(LinkedHashMap.class); p.apply("readFromSalesforcePlugin", readTransform); {{< /highlight >}} {{< highlight java >}} ServiceNowSourceConfig pluginConfig = new ConfigWrapper<>(ServiceNowSourceConfig.class).withParams(pluginConfigParams).build(); CdapIO<NullWritable, StructuredRecord> readTransform = CdapIO.<NullWritable, StructuredRecord>read() .withCdapPluginClass(ServiceNowSource.class) .withPluginConfig(pluginConfig) .withKeyClass(NullWritable.class) .withValueClass(StructuredRecord.class); p.apply("readFromServiceNowPlugin", readTransform); {{< /highlight >}} {{< highlight java >}} ZendeskBatchSourceConfig pluginConfig = new ConfigWrapper<>(ZendeskBatchSourceConfig.class).withParams(pluginConfigParams).build(); CdapIO<NullWritable, StructuredRecord> readTransform = CdapIO.<NullWritable, StructuredRecord>read() .withCdapPluginClass(ZendeskBatchSource.class) .withPluginConfig(pluginConfig) .withKeyClass(NullWritable.class) .withValueClass(StructuredRecord.class); p.apply("readFromZendeskPlugin", readTransform); {{< /highlight >}} To learn more please check out . In order to write to CDAP plugin you will need to pass: `Key` and `Value` classes. You will need to check if these classes have a Beam Coder available. `locksDirPath`, which is locks directory path where locks will be stored. This parameter is needed for Hadoop External Synchronization (mechanism for acquiring locks related to the write

`PluginConfig` object with parameters for certain CDAP plugin. You can easily build `PluginConfig` object using `ConfigWrapper` class by specifying: Class of the needed `PluginConfig`. `Map<String, Object>` parameters map for corresponding CDAP plugin. For example: {{< highlight java >}} MyPluginConfig pluginConfig = new ConfigWrapper<>(MyPluginConfig.class).withParams(pluginConfigParams).build(); {{< /highlight >}} Some CDAP plugins are already supported and can be used just by plugin class name. For example: {{< highlight java >}} CdapIO.Write<NullWritable, String> readTransform = CdapIO.<NullWritable, String>write() .withCdapPluginClass(HubspotBatchSink.class) .withPluginConfig(pluginConfig) .withKeyClass(NullWritable.class) .withValueClass(String.class) .withLocksDirPath(locksDirPath); p.apply("write", writeTransform); {{< /highlight >}} If CDAP plugin is not supported by plugin class name, you can easily build `Plugin` object by passing the following parameters: Class of CDAP plugin. The `OutputFormat` class used to connect to your CDAP plugin of choice. The `OutputFormatProvider` class used to provide `OutputFormat`. Then you will be able to pass this `Plugin` object to `CdapIO`. For example: {{< highlight java >}} CdapIO.Write<String, String> writeTransform = CdapIO.<String, String>write() .withCdapPlugin( Plugin.createBatch( MyCdapPlugin.class, MyOutputFormat.class, MyOutputFormatProvider.class)) .withPluginConfig(pluginConfig) .withKeyClass(String.class) .withValueClass(String.class) .withLocksDirPath(locksDirPath); p.apply("write", writeTransform); {{< /highlight >}} {{< highlight java >}} SinkHubspotConfig pluginConfig = new ConfigWrapper<>(SinkHubspotConfig.class).withParams(pluginConfigParams).build(); CdapIO<NullWritable, String> writeTransform = CdapIO.<NullWritable, String>write() .withCdapPluginClass(pluginClass) .withPluginConfig(pluginConfig) .withKeyClass(NullWritable.class) .withValueClass(String.class) .withLocksDirPath(locksDirPath); p.apply("writeToHubspotPlugin", writeTransform); {{< /highlight >}} {{< highlight java >}} SalesforceSinkConfig pluginConfig = new ConfigWrapper<>(SalesforceSinkConfig.class).withParams(pluginConfigParams).build(); CdapIO<NullWritable, CSVRecord> writeTransform = CdapIO.<NullWritable, CSVRecord>write() .withCdapPluginClass(pluginClass) .withPluginConfig(pluginConfig) .withKeyClass(NullWritable.class) .withValueClass(CSVRecord.class) .withLocksDirPath(locksDirPath); p.apply("writeToSalesforcePlugin", writeTransform); {{< /highlight >}} To learn more please check out . In order to read from CDAP plugin you will need to pass: `Key` and `Value` classes. You will need to check if these classes have a Beam Coder available. `PluginConfig` object with parameters for certain CDAP plugin. You can easily build `PluginConfig` object using `ConfigWrapper` class by specifying: Class of the needed `PluginConfig`. `Map<String, Object>` parameters map for corresponding CDAP plugin. For example: {{< highlight java >}} MyPluginConfig pluginConfig = new ConfigWrapper<>(MyPluginConfig.class).withParams(pluginConfigParams).build(); {{< /highlight >}} Some CDAP plugins are already supported and can be used just by plugin class name. For example: {{< highlight java >}} CdapIO.Read<String, String> readTransform = CdapIO.<String, String>read() .withCdapPluginClass(MyStreamingPlugin.class) .withPluginConfig(pluginConfig) .withKeyClass(NullWritable.class) .withValueClass(String.class); p.apply("read", readTransform); {{< /highlight >}} If CDAP plugin is not supported by plugin class name, you can easily build `Plugin` object by passing the following parameters: Class of CDAP Streaming plugin. `getOffsetFn`, which is `SerializableFunction` that defines how to get `Long` record offset from a record. `receiverClass`, which is Spark (v 2.4) `Receiver` class associated with CDAP plugin. (Optionally) `getReceiverArgsFromConfigFn`, which is `SerializableFunction` that defines how to get constructor arguments for Spark `Receiver` using `PluginConfig` object. Then you will be able to pass this `Plugin` object to `CdapIO`. For example: {{< highlight java >}} CdapIO.Read<String, String> readTransform = CdapIO.<String, String>read() .withCdapPlugin( Plugin.createStreaming( MyStreamingPlugin.class, myGetOffsetFn, MyReceiver.class, myGetReceiverArgsFromConfigFn)) .withPluginConfig(pluginConfig) .withKeyClass(NullWritable.class) .withValueClass(String.class); p.apply("read", readTransform); {{< /highlight >}} Optionally you can pass the following optional parameters: `pullFrequencySec`, which is delay in seconds between polling for new records updates. `startOffset`, which is inclusive start offset from which the reading should be started. For example: {{< highlight java >}} CdapIO.Read<String, String> readTransform = CdapIO.<String, String>read() .withCdapPluginClass(MyStreamingPlugin.class) .withPluginConfig(pluginConfig) .withKeyClass(NullWritable.class) .withValueClass(String.class) .withPullFrequencySec(1L) .withStartOffset(1L); p.apply("read", readTransform); {{< /highlight >}} {{< highlight java >}} HubspotStreamingSourceConfig pluginConfig = new ConfigWrapper<>(HubspotStreamingSourceConfig.class) .withParams(pluginConfigParams).build(); CdapIO.Read<NullWritable, String> readTransform = CdapIO.<NullWritable, String>read() .withCdapPlugin( Plugin.createStreaming( HubspotStreamingSource.class, GetOffsetUtils.getOffsetFnForHubspot(), HubspotReceiver.class)) .withPluginConfig(pluginConfig) .withKeyClass(NullWritable.class) .withValueClass(String.class); p.apply("readFromHubspotPlugin", readTransform); {{< /highlight >}} {{< highlight java >}} SalesforceStreamingSourceConfig pluginConfig = new ConfigWrapper<>(SalesforceStreamingSourceConfig.class) .withParams(pluginConfigParams).build(); CdapIO.Read<NullWritable, String> readTransform = CdapIO.<NullWritable, String>read() .withCdapPlugin( Plugin.createStreaming( SalesforceStreamingSource.class, GetOffsetUtils.getOffsetFnForSalesforce(), SalesforceReceiver.class, config -> { SalesforceStreamingSourceConfig salesforceConfig = SalesforceStreamingSourceConfig) config; return new Object[] { salesforceConfig.getAuthenticatorCredentials(), salesforceConfig.getPushTopicName() }; })) .withPluginConfig(pluginConfig) .withKeyClass(NullWritable.class) .withValueClass(String.class); p.apply("readFromSalesforcePlugin", readTransform); {{< /highlight >}} To learn more please check out .