USER DOCUMENTATION Home Keboola Overview Getting Started Tutorial Components Extractors Communication Databases Azure Storage Table CosmosDB Google BigQuery MongoDB SQL Databases Marketing/Sales Social Storage Other Writers Applications Data Templates Development Branches IP Addresses Data Catalog Storage Orchestrations Transformations Management Home Components Extractors Databases Database Extractors SQL Databases NoSQL Databases Connecting to Database Incremental Fetching Extractors import data from external sources and integrate it to the Keboola Connection environment. SQL Databases There are a number of extractors for SQL databases: Cloudera Impala Firebird IBM DB2 Microsoft SQL Server MySQL Oracle PostgreSQL Redshift Snowflake Teradata They are all configured in the same way and have an advanced mode. Their basic configuration is also part of the Tutorial - Loading Data with Database Extractor. NoSQL Databases Unlike the extractors for SQL databases, extractors for NoSQL databases require a different configuration (except the BigQuery extractor for the BigQuery database, which is quite similar to SQL databases and also supports the advanced mode): MongoDB extractor for the MongoDB database and the CosmosDB MongoDB API. CosmosDB extractor for the CosmosDB SQL API. Azure Storage Table extractor for the Azure Table storage and the Cosmos DB Table API. Connecting to Database The connection to your internal database must be well secured. If you, or your system administrator, want to avoid exposing your database server to the internet, we highly recommend setting up an SSH Tunnel for the connection. A Secure Shell (SSH) tunnel consists of an encrypted connection created through an SSH protocol connection. You can set up this tunnel to connect to your database server located in your private network that you do not want to be accessed directly from the internet. The SSH connection is encrypted and uses a public-private key pair for user authorization. Find detailed instructions for setting up an SSH tunnel in the developer documentation. While setting up an SSH tunnel requires some work, it is the most reliable and secure option for connecting to your database server. Incremental Fetching Some database extractors support the feature of Incremental Fetching. Incremental fetching allows to dramatically reduce the amount of data loaded from the source database server if the nature of the database permits. Incremental fetching can be used when the source table contains an ordering (ordinal) column and no rows are deleted (or the deletions are not important). There are two typical scenarios: A table to which rows are only added, and the rows have numeric and raising ID. A table in which rows are added and modified, and it contains a column with last modification time. To enable incremental fetching, you have to specify the ordering column. When incremental fetching is enabled, the rows in the table are extracted in the order specified by the ordering column. The last fetched value is recorded in the configuration state. When the extraction runs again, only new rows are fetched from the sourced database. To configure incremental fetching, go to table details: Then select the ordering column: The last fetched value is displayed in the configuration: The rows are fetched from the source table including the last fetched value. Therefore it is ideal to have the ordering column set as a primary key so that you don’t receive duplicated rows in the Storage table. In case you need to fetch the entire table, you can clear the stored value. This incremental fetching feature is related to incremental loading. While not required, it is recommended to turn on incremental loading when fetching data incrementally, otherwise the table in Storage will contain only newly added rows. This may sound like a good idea when you want to process only the newly added rows. In that case, however, you should do so using incremental processing. The advantage of using incremental processing over having only newly added rows in a Storage table is that the table contains all loaded data and it is not necessary to synchronize extraction and processing. © 2021 Keboola Provide feedback