Repository: keboola/connection-docs
Pull request #746: Added AI Flow Builder + removed Twitter extractor
Author: login: pavel-synek
Created: 2025-03-19T14:41:53Z | Merged: 2025-03-27T10:39:27Z

Twitter extractor removal is based on deprecation of the component : https://keboola.atlassian.net/browse/SUPPORT-11010


Diff:
diff --git a/_data/navigation.yml b/_data/navigation.yml
index df34d5564..4e1785821 100644
--- a/_data/navigation.yml
+++ b/_data/navigation.yml
@@ -239,9 +239,6 @@ items:
               - url: /components/extractors/social/instagram/
                 title: Instagram
 
-              - url: /components/extractors/social/twitter/
-                title: Twitter
-
               - url: /components/extractors/social/youtube/
                 title: YouTube Data API
 
@@ -593,6 +590,9 @@ items:
 
   - url: /flows/
     title: Flows
+    items:
+      - url: /flows/ai-flow-builder/
+        title: AI Flow Builder
 
   - url: /orchestrator/
     title: Orchestrations
diff --git a/components/extractors/database/mysql/index.md b/components/extractors/database/mysql/index.md
index 63cc05a7e..41f6a19ea 100644
--- a/components/extractors/database/mysql/index.md
+++ b/components/extractors/database/mysql/index.md
@@ -538,7 +538,7 @@ CREATE TABLE debezium_signal (id VARCHAR(42) PRIMARY KEY, type VARCHAR(32) NOT N
 #### SSH tunnel
 
 You may opt to use an SSH Tunnel to secure your connection. Find detailed instructions for setting up an SSH tunnel in
-the [developer documentation](https://developers.keboola.com/integrate/database/. While setting up an SSH tunnel requires some work, it is the most reliable and secure
+the [developer documentation](https://developers.keboola.com/integrate/database/). While setting up an SSH tunnel requires some work, it is the most reliable and secure
 option for connecting to your database server.
 
 ### Data Source
diff --git a/components/extractors/social/twitter/index.md b/components/extractors/social/twitter/index.md
deleted file mode 100644
index 0d8737131..000000000
--- a/components/extractors/social/twitter/index.md
+++ /dev/null
@@ -1,144 +0,0 @@
----
-title: Twitter
-permalink: /components/extractors/social/twitter/
-redirect_from:
-    - /extractors/social/twitter/
----
-
-* TOC
-{:toc}
-
-The Twitter data source connector, as the name implies, fetches data from [Twitter](https://twitter.com/).
-
-## Configuration
-[Create a new configuration](/components/#creating-component-configuration) of the **Twitter** connector.
-Then click **Authorize Account** to [authorize the configuration](/components/#authorization). 
-When finished setting up authorization, click **Continue** to move to the next tab, and start selecting data.
-
-{: .image-popup}
-![Twitter Data Selection](/components/extractors/social/twitter/twitter-1.png)
-
-## Select Data
-To configure what you want to fetch from Twitter, use the other four tabs displayed in the above pop-up window.
-It is okay to leave any of the steps blank.
-
- - **User Timeline** -- Returns a collection of the most recent tweets posted by the user indicated by the *Screen Name*.
-
-   {: .image-popup}
-![Twitter Screen Name](/components/extractors/social/twitter/screen-name.png)
-
-   - [Limited to last 3,200 tweets](https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline)
- - **Mentions** -- Returns mentions for the authenticating user.
-   - [Limited to last 800 mentions of the account](https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-mentions_timeline)
- - **Followers List** -- Returns a list of users following the specified user by their *Screen Name*.
- - **Search** -- Searches tweets. Read [more about search queries](https://developer.twitter.com/en/docs/tweets/search/overview).
-    - The Twitter Search API searches against a sampling of recent tweets published within the past 7 days.
-
-## Extraction Output Tables
-
-The connector output is always **incremental**, and **primary keys** are set for all tables
-to append new rows and update the already existing ones with each execution.
-
-The following are the Twitter connector output tables:
-
-### 1 -- Tweets
-
-The Tweets table contains:
-
-- Tweets from the `User Timeline`, `Mentions` or `Search` tabs
-- Tweets attached to one of the above mentioned tweets. These are usually the original tweets that were retweeted and were referenced in the `retweeted_status_id` of the retweet.
-
-The origin of a tweet is distinguished by a special `keboola_source` column with the following values for each source:
-
-  - `userTimeline` -- `User Timeline`
-  - `mentions` -- `Mentions`
-  - `search` -- `Search`
-
-The table columns mirror [the structure of the Twitter Tweet entity](https://developer.twitter.com/en/docs/tweets/data-dictionary/overview/tweet-object):
-
-| Column | Description |
-| `id` [PK] | The integer representation of the unique identifier for the tweet. |
-| `created_at` | UTC time when this tweet was created.|
-| `favorite_count` |  Indicates approximately how many times this tweet has been “liked” by Twitter users. |
-| `in_reply_to_screen_name` | If the represented tweet is a reply, this field will contain the screen name of the original Tweet’s author. |
-| `in_reply_to_status_id` | If the represented tweet is a reply, this field will contain the integer representation of the original tweet’s ID. |
-| `in_reply_to_user_id` | If the represented tweet is a reply, this field will contain the integer representation of the original tweet’s author ID. This will not necessarily always be the user directly mentioned in the tweet. |
-| `lang` | When present, it indicates a BCP 47 language identifier corresponding to the machine-detected language of the tweet text, or “und” if no language could be detected. |
-| `quoted_status_id` | This field only surfaces when the tweet is a quote tweet. This field contains the integer value tweet ID of the quoted tweet. |
-| `retweet_count` | Number of times this tweet has been retweeted. This field is no longer capped at 99 and will not turn into a string for “100+” |
-| `retweeted_status_id` | ID of the original tweet. Referenced tweet should be also in this table. |
-| `source` | Utility used to post the tweet as an HTML-formatted string. |
-| `text` | The actual UTF-8 text of the status update. |
-| `truncated` | Indicates whether the value of the text parameter was truncated, for example, as a result of a retweet exceeding the 280 character tweet length. Truncated text will end in ellipsis, like this ... Since Twitter now rejects long tweets vs truncating them, the large majority of tweets will have this set to false. |
-| `user_id` | Reference to the user who posted this tweet. The referenced user can be found in `users` with `id` equals to `user_id`. |
-| `withheld_copyright` | When present and set to “true”, it indicates that this piece of content has been withheld due to a DMCA complaint. |
-| `keboola_source` | Source of the tweet - `userTimeline`, `mentions` or `search`. When the same tweet is the source of more targets (for instance, search and mentions), only one (later imported) source is set.
-
-### 2 -- Users
-
-The Users table contains:
-
-- Users from the `Followers List` tab
-- Users attached to tweets
-
-The origin of a user is distinguished by a special `keboola_source` column with the following values for each source:
-
-  - `Followers List` - `followersList`
-  - `userTimeline`, `mentions`, `search` - Derived from associated tweet
-
-The table columns mirror [the structure of Twitter User entity](https://developer.twitter.com/en/docs/tweets/data-dictionary/overview/user-object):
-
-| Column | Description |
-| `id` [PK] | The integer representation of the unique identifier for this User. |
-| `created_at` | The UTC datetime that the user account was created on Twitter. |
-| `description` | The user-defined UTF-8 string describing their account. |
-| `favourites_count` | The number of tweets this user has favored in the account’s lifetime. British spelling is used in the field name for historical reasons. |
-| `followers_count` | The number of followers this account currently has. Under certain conditions of duress, this field will temporarily indicate “0.” |
-| `friends_count` | The number of users this account is following (AKA their “followings”). Under certain conditions of duress, this field will temporarily indicate “0.” |
-| `lang` | The BCP 47 code for the user’s self-declared user interface language. May or may not have anything to do with the content of their tweets. |
-| `name` | The name of the user, as they’ve defined it. Not necessarily a person’s name. Typically capped at 20 characters, but subject to change. |
-| `screen_name` | The screen name, handle, or alias that this user identifies themselves with. screen_names are unique but subject to change. Use id_str as a user identifier whenever possible. Typically a maximum of 15 characters long, but some historical accounts may exist with longer names. |
-| `statuses_count` | The number of tweets (including retweets) issued by the user. |
-| `keboola_source` | Source of the user - `followersList`, `userTimeline`, `mentions` or `search`. When the same tweet is the source of more targets (e.g. search and mentions), only one (later imported) source is set.
-
-### 3 -- tweets-user-mentions
-
-This table contains the users mentioned in tweets.
-
-| Column | Description |
-| `user_id` [PK] | ID of [user](#table---users) |
-| `tweets_pk` [PK] |  ID of [tweet](#table---tweets) |
-| `name` | The name of the user, as they’ve defined it. Not necessarily a person’s name. Typically capped at 20 characters, but subject to change. |
-| `screen_name` | The screen name, handle, or alias that this user identifies themselves with. screen_names are unique but subject to change. Use id_str as a user identifier whenever possible. Typically a maximum of 15 characters long, but some historical accounts may exist with longer names.  |
-
-### 4 -- tweets-urls
-
-This table contains the URLs included in the text of a tweet.
-
-| Column | Description |
-| `tweets_pk` [PK] |  ID of [tweet](#table---tweets) |
-| `url` [PK] | Wrapped URL, corresponding to the value embedded directly into the raw tweet text. |
-| `display_url` | Version of the URL to display to clients. |
-| `expanded_url` | Expanded version of display_url. |
-
-### 5 -- tweets-hashtags
-
-This table contains the hashtags present in tweets.
-
-| Column | Description |
-| `tweets_pk` [PK] |  ID of [tweet](#table---tweets) |
-| `text` [PK] | Name of the hashtag, minus the leading ‘#’ character. |
-
-## Usage Limits
-
-This data source connector is designed to download up to
-
-- 3,200 most recent user tweets (User Timeline)
--  800 mentions (Mentions)
-- 3,000 followers (Followers List)
-- ~3,000 tweets (Tweet Search)
-
-in one execution. Following executions will incrementally add newer data.
-
-
-These limits follow the [Twitter API Rate Limits](https://developer.twitter.com/en/docs/basics/rate-limiting).
diff --git a/components/extractors/social/twitter/screen-name.png b/components/extractors/social/twitter/screen-name.png
deleted file mode 100644
index 0f9f54604..000000000
Binary files a/components/extractors/social/twitter/screen-name.png and /dev/null differ
diff --git a/components/extractors/social/twitter/twitter-1.png b/components/extractors/social/twitter/twitter-1.png
deleted file mode 100644
index 9db64d6ed..000000000
Binary files a/components/extractors/social/twitter/twitter-1.png and /dev/null differ
diff --git a/flows/ai-flow-builder/index.md b/flows/ai-flow-builder/index.md
new file mode 100644
index 000000000..dbb69fe73
--- /dev/null
+++ b/flows/ai-flow-builder/index.md
@@ -0,0 +1,71 @@
+---
+title: AI Flow Builder
+permalink: /ai-flow-builder/
+---
+
+* TOC
+{:toc}
+
+# AI Flow Builder
+
+AI Flow Builder is an experimental AI assistant that helps you create a simple 3-step ETL flow from end to end based on your intent. It is available exclusively in **Pay-As-You-Go** projects, with the primary purpose of jumpstarting the first Flow creation. The feature can be enabled/disabled from the Project Settings.
+
+This version is offered **free of charge** as it is subject to change, and we plan to expand its functionalities in the future.
+
+## Functionality
+
+The AI-assisted Flow Builder process is divided into **three logical episodes**.
+
+### 1. Defining the Flow Structure Based on Your Intent
+You provide an intent, e.g., _"I would like to analyze my marketing campaigns,"_ and the assistant will ask additional questions about the data sources and desired destination to build the Flow structure. Once you confirm the suggested sequence of steps (components), the Flow is created with **empty configurations**, and the configuration process can begin.
+
+### 2. Flow Configuration
+At this stage, you gradually configure the components, starting with the first phase of the Flow (**Extract**). When you click on a component configuration to open the **configuration detail view**, the assistant will guide you through the setup.
+
+Once the component configuration is completed, you click the **"BACK TO FLOW"** button, and the component execution starts automatically. During this period, you can **monitor the progress** or proceed with configuring the remaining components within the Flow phase.
+
+If any errors occur, you will be notified, and navigating to the failed component configuration will prompt the assistant to help resolve the issue.
+
+**Important:** The AI Flow Builder **does not allow** proceeding to the next Flow phase until all components are fully configured and their execution is completed successfully (indicated by a **green checkmark**).
+
+### 3. Previewing Results
+The output of a component can be previewed by clicking the **"EXPLORE DATA"** icon located at the bottom-right corner of the component icon.
+
+### AI Flow Builder Drafts
+You may choose to leave the AI Flow Builder before completing the Flow configuration. In such cases, the Flow is labeled as a **"DRAFT"**, allowing you to return later and finish the setup.
+
+## Transformation Phase
+Once the **Extract** phase is completed, you can proceed to configure the **SQL transformation**. All data from the previous phase's outputs will be **automatically available** in the Transformation input mapping.
+
+The assistant will guide you through the **SQL code creation** process, allowing you to ask additional questions and test the generated query until you are satisfied.
+
+Once the transformation is completed, the outputs of this phase will be **automatically used** in the **destination phase** if configured.
+
+## Destination (Load) Phase
+If a **destination component** was specified during the initial Flow setup, it will be included in the Flow. The setup process for this phase is similar to previous phases—you configure the component, and once done, it will execute when you return to the Flow.
+
+## Flow Publishing
+Once **all jobs** in all Flow phases are successfully completed, you can click the **"PUBLISH"** button. This finalizes the Flow, and you are redirected to the standard Flow Builder, where you can set up **scheduling** or make additional modifications as needed.
+
+**Important:** Once the Flow is published, the **AI Flow Builder can no longer be used** for that Flow.
+
+## Limitations
+
+### 1. **Only 3-Step (Phase) Flows Are Supported**
+- The AI Flow Builder supports Flows with up to **three phases**: **Extract, Transform, and Load**.
+- The **Extract** phase may contain **multiple connectors**.
+- The **Transform** phase supports **only one SQL transformation configuration**.
+- The **Load** phase is **optional** and contains a destination component. If omitted, the transformed data will be stored in **Keboola Storage**.
+- After publishing the AI-generated Flow, you can freely edit it using the **standard Flow Builder**, adding additional phases or configurations.
+
+### 2. **Only AI-Assisted SQL Transformations Are Supported**
+- In the **Data Processing phase**, only **AI-assisted SQL transformations** are currently supported.
+
+### 3. **AI Flow Builder Cannot Be Used on Existing Flows**
+- The AI Flow Builder is designed to create new Flows **from scratch**. It **cannot** be used on existing Flows.
+
+### 4. **Scheduling Can Be Set Up Only After Publishing**
+- Flow scheduling can be configured **only after** the AI Flow Builder process is **completed and published** as a standard Flow.
+
+### 5. **Flow Structure Cannot Be Changed After Confirmation**
+- Once you **confirm the Flow structure and components**, **it cannot be modified later** within the AI Flow Builder. Any further modifications must be made in the standard Flow Builder.


Reviews:
user: login: pivnicek
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T13:52:34Z
user: login: pavel-synek
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-26T15:02:54Z
user: login: davidesner
body: good to me
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-27T09:14:03Z

Review comments:
diff_hunk: @@ -239,9 +239,6 @@ items:
               - url: /components/extractors/social/instagram/
                 title: Instagram
 
-              - url: /components/extractors/social/twitter/
-                title: Twitter
path: _data/navigation.yml
user: login: pivnicek
body: @pavel-synek please also remove the source materials, I think you can delete that whole folder
created_at: 2025-03-26T13:52:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -239,9 +239,6 @@ items:
               - url: /components/extractors/social/instagram/
                 title: Instagram
 
-              - url: /components/extractors/social/twitter/
-                title: Twitter
path: _data/navigation.yml
user: login: pavel-synek
body: @pivnicek  Good catch. Thanks. Done. Can you see the deletion within the PR, please? 
created_at: 2025-03-26T15:02:54Z
author_association: COLLABORATOR

PR comments:
user: login: pavel-synek
created_at: 2025-03-26T13:45:35Z
body: Fixed link in the MySQL extractor Docs according to the ticket: https://keboola.atlassian.net/browse/SUPPORT-11379
author_association: COLLABORATOR
user: login: davidesner
created_at: 2025-03-27T09:13:21Z
body: @pavel-synek I added some modifications. 

- Structure updates so it looks better visually
- Beta warning
- Warning box about limitations
- Mentioning the AI flow builder in the Flow index page
- Fixing the AI Flow Builder link so it doesn't throw 404
author_association: CONTRIBUTOR