Repository: google/dwh-migration-tools
Pull request #766: [b/390590500] Airflow dates params
Author: login: vladislav-sidorovich
Created: 2025-02-28T12:59:05Z | Merged: 2025-03-25T09:35:31Z

Add date range support to airflow connector.

Usage:

1. `--lookback-days=x` -> date range: `[now-x, now)`
2. `--start-date=x --end-date=y` -> date range: `[x, y)`
3. `-start-date=x --lookback-days=y` -> date range: `[x-y, x)`

If date range is provided, sql queries will be generate for each day in the interval.  


Diff:
diff --git a/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/ConnectorArguments.java b/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/ConnectorArguments.java
index fbfdc65ca..a58baa6de 100644
--- a/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/ConnectorArguments.java
+++ b/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/ConnectorArguments.java
@@ -91,6 +91,8 @@ public class ConnectorArguments extends DefaultArguments {
   public static final int OPT_PORT_ORDER = 200;
   public static final String OPT_USER = "user";
   public static final String OPT_PASSWORD = "password";
+  public static final String OPT_START_DATE = "start-date";
+  public static final String OPT_END_DATE = "end-date";
 
   public static final String OPT_CLUSTER = "cluster";
   public static final String OPT_ROLE = "role";
@@ -228,6 +230,26 @@ public class ConnectorArguments extends DefaultArguments {
           .withOptionalArg()
           .describedAs("sekr1t");
 
+  private final OptionSpec<ZonedDateTime> optionStartDate =
+      parser
+          .accepts(
+              OPT_START_DATE,
+              "Inclusive start date for data to export, value will be truncated to hour")
+          .withOptionalArg()
+          .ofType(Date.class)
+          .withValuesConvertedBy(ZonedParser.withDefaultPattern(DayOffset.START_OF_DAY))
+          .describedAs("2001-01-15[ 00:00:00.[000]]");
+
+  private final OptionSpec<ZonedDateTime> optionEndDate =
+      parser
+          .accepts(
+              OPT_END_DATE,
+              "Exclusive end date for data to export, value will be truncated to hour")
+          .withOptionalArg()
+          .ofType(Date.class)
+          .withValuesConvertedBy(ZonedParser.withDefaultPattern(DayOffset.START_OF_DAY))
+          .describedAs("2001-01-15[ 00:00:00.[000]]");
+
   private final OptionSpec<String> optionCluster =
       parser
           .accepts(OPT_CLUSTER, "Cluster name to dump metadata")
@@ -309,7 +331,7 @@ public class ConnectorArguments extends DefaultArguments {
           .withOptionalArg()
           .ofType(Date.class)
           .withValuesConvertedBy(ZonedParser.withDefaultPattern(DayOffset.START_OF_DAY))
-          .describedAs("2001-01-01[ 00:00:00.[000]]");
+          .describedAs("2001-01-15[ 00:00:00.[000]]");
   private final OptionSpec<ZonedDateTime> optionQueryLogEnd =
       parser
           .accepts(
@@ -318,7 +340,7 @@ public class ConnectorArguments extends DefaultArguments {
           .withOptionalArg()
           .ofType(Date.class)
           .withValuesConvertedBy(ZonedParser.withDefaultPattern(DayOffset.END_OF_DAY))
-          .describedAs("2001-01-01[ 00:00:00.[000]]");
+          .describedAs("2001-01-15[ 00:00:00.[000]]");
 
   // This is intentionally NOT provided as a default value to the optionQueryLogEnd OptionSpec,
   // because some callers
@@ -889,6 +911,20 @@ public Integer getQueryLogDays() {
     return getOptions().valueOf(optionQueryLogDays);
   }
 
+  @CheckForNull
+  public ZonedDateTime getStartDate() {
+    return getOptions().valueOf(optionStartDate);
+  }
+
+  public ZonedDateTime getStartDate(ZonedDateTime defaultTime) {
+    return firstNonNull(getStartDate(), defaultTime);
+  }
+
+  @CheckForNull
+  public ZonedDateTime getEndDate() {
+    return getOptions().valueOf(optionEndDate);
+  }
+
   public Duration getQueryLogRotationFrequency() {
     return RotationFrequencyConverter.convert(
         getOptions().valueOf(optionQueryLogRotationFrequency));
diff --git a/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/annotations/RespectsArgumentQueryLogDays.java b/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/annotations/RespectsArgumentQueryLogDays.java
index 5c995568e..fde0597fe 100644
--- a/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/annotations/RespectsArgumentQueryLogDays.java
+++ b/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/annotations/RespectsArgumentQueryLogDays.java
@@ -35,5 +35,5 @@
     description = RespectsArgumentQueryLogDays.DESCRIPTION)
 public @interface RespectsArgumentQueryLogDays {
 
-  public static final String DESCRIPTION = "The number of days of query history to dump.";
+  String DESCRIPTION = "The number of days of query history to dump.";
 }
diff --git a/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/annotations/RespectsArgumentQueryLogEnd.java b/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/annotations/RespectsArgumentQueryLogEnd.java
index ad30c7d3b..dfa18d3ab 100644
--- a/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/annotations/RespectsArgumentQueryLogEnd.java
+++ b/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/annotations/RespectsArgumentQueryLogEnd.java
@@ -35,5 +35,5 @@
     description = RespectsArgumentQueryLogEnd.DESCRIPTION)
 public @interface RespectsArgumentQueryLogEnd {
 
-  public static final String DESCRIPTION = "End date for query history to dump";
+  String DESCRIPTION = "End date for query history to dump";
 }
diff --git a/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/annotations/RespectsArgumentQueryLogStart.java b/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/annotations/RespectsArgumentQueryLogStart.java
index 4b42bb733..ad386c3b8 100644
--- a/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/annotations/RespectsArgumentQueryLogStart.java
+++ b/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/annotations/RespectsArgumentQueryLogStart.java
@@ -35,5 +35,5 @@
     description = RespectsArgumentQueryLogStart.DESCRIPTION)
 public @interface RespectsArgumentQueryLogStart {
 
-  public static final String DESCRIPTION = "Start date for query history to dump";
+  String DESCRIPTION = "Start date for query history to dump";
 }
diff --git a/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/AbstractJdbcConnector.java b/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/AbstractJdbcConnector.java
index 287c3f8ba..6c94823d0 100644
--- a/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/AbstractJdbcConnector.java
+++ b/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/AbstractJdbcConnector.java
@@ -28,6 +28,9 @@
 import java.security.PrivilegedExceptionAction;
 import java.sql.Driver;
 import java.sql.SQLException;
+import java.time.ZoneOffset;
+import java.time.ZonedDateTime;
+import java.time.format.DateTimeFormatter;
 import java.util.ArrayList;
 import java.util.Arrays;
 import java.util.List;
@@ -39,14 +42,18 @@
 
 /** @author shevek */
 public abstract class AbstractJdbcConnector extends AbstractConnector {
-
-  @SuppressWarnings("UnusedVariable")
   private static final Logger logger = LoggerFactory.getLogger(AbstractJdbcConnector.class);
+  private static final DateTimeFormatter SQL_FORMAT =
+      DateTimeFormatter.ISO_OFFSET_DATE_TIME.withZone(ZoneOffset.UTC);
 
   public AbstractJdbcConnector(@Nonnull String name) {
     super(name);
   }
 
+  protected static String dateToSqlFormat(ZonedDateTime dateTime) {
+    return SQL_FORMAT.format(dateTime);
+  }
+
   @Nonnull
   private static ClassLoader newDriverParentClassLoader() throws PrivilegedActionException {
     return AccessController.doPrivileged(
diff --git a/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/ZonedIntervalIterable.java b/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/ZonedIntervalIterable.java
index 94c508b21..aa903738a 100644
--- a/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/ZonedIntervalIterable.java
+++ b/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/ZonedIntervalIterable.java
@@ -23,30 +23,25 @@
 import java.util.Iterator;
 import javax.annotation.Nonnull;
 import org.apache.commons.lang3.time.DurationFormatUtils;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
 
 /** @author shevek */
 public class ZonedIntervalIterable implements Iterable<ZonedInterval> {
-
-  @SuppressWarnings("UnusedVariable")
-  private static final Logger logger = LoggerFactory.getLogger(ZonedIntervalIterable.class);
-
   private final ZonedDateTime start;
   private final ZonedDateTime end;
-  private final Duration duration;
+  private final Duration intervalDuration;
 
-  /* pp */ ZonedIntervalIterable(
+  ZonedIntervalIterable(
       @Nonnull ZonedDateTime start,
       @Nonnull ZonedDateTime end,
-      @Nonnull Duration duration,
+      @Nonnull Duration intervalDuration,
       IntervalExpander expander) {
-    this.duration = Preconditions.checkNotNull(duration, "Duration was null.");
+    this.intervalDuration =
+        Preconditions.checkNotNull(intervalDuration, "Interval duration was null.");
     Preconditions.checkNotNull(start, "Start was null.");
     Preconditions.checkNotNull(end, "End was null.");
 
     Preconditions.checkState(
-        start.isBefore(end), "Start date %s must precede end date %s", start, end);
+        start.isBefore(end), "Start date [%s] must precede end date [%s]", start, end);
 
     ZonedInterval expandedInterval = expander.apply(new ZonedInterval(start, end));
 
@@ -65,15 +60,15 @@ public ZonedDateTime getEnd() {
   }
 
   @Nonnull
-  public Duration getDuration() {
-    return duration;
+  public Duration getIntervalDuration() {
+    return intervalDuration;
   }
 
-  private class Itr extends AbstractIterator<ZonedInterval> {
+  private class DatesIterator extends AbstractIterator<ZonedInterval> {
 
     private ZonedDateTime current;
 
-    public Itr() {
+    public DatesIterator() {
       this.current = start;
     }
 
@@ -81,7 +76,7 @@ public Itr() {
     protected ZonedInterval computeNext() {
       if (current.isEqual(end) || current.isAfter(end)) return endOfData();
 
-      ZonedDateTime next = current.plus(duration);
+      ZonedDateTime next = current.plus(intervalDuration);
       if (next.isAfter(end)) return endOfData();
 
       ZonedInterval result = new ZonedInterval(current, next);
@@ -93,13 +88,15 @@ protected ZonedInterval computeNext() {
   @Nonnull
   @Override
   public Iterator<ZonedInterval> iterator() {
-    return new Itr();
+    return new DatesIterator();
   }
 
   @Override
   public String toString() {
     return String.format(
-        "from %s to %s every %s",
-        start, end, DurationFormatUtils.formatDurationWords(duration.toMillis(), true, true));
+        "from [%s] to [%s] every [%s]ms",
+        start,
+        end,
+        DurationFormatUtils.formatDurationWords(intervalDuration.toMillis(), true, true));
   }
 }
diff --git a/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/ZonedIntervalIterableGenerator.java b/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/ZonedIntervalIterableGenerator.java
index b9714d846..2511f1b06 100644
--- a/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/ZonedIntervalIterableGenerator.java
+++ b/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/ZonedIntervalIterableGenerator.java
@@ -56,6 +56,17 @@ public class ZonedIntervalIterableGenerator {
     return forTimeUnitsUntil(ZonedDateTime.now(ZoneOffset.UTC), unitCount, duration, expander);
   }
 
+  @Nonnull
+  public static ZonedIntervalIterable forDateRangeWithIntervalDuration(
+      ZonedDateTime startDate, ZonedDateTime endDate, Duration intervalDuration)
+      throws MetadataDumperUsageException {
+    return new ZonedIntervalIterable(
+        startDate,
+        endDate,
+        intervalDuration,
+        IntervalExpander.createBasedOnDuration(intervalDuration));
+  }
+
   /**
    * Builds a ZonedIntervalIterable from connector arguments, the intervals will all be one hour
    * long ({@link ChronoUnit#HOURS}) and have an inclusive starting datetime and exclusive ending
diff --git a/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java b/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
index e46996315..c6e836f94 100644
--- a/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
+++ b/dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
@@ -30,6 +30,9 @@
 import com.google.edwmigration.dumper.application.dumper.connector.AbstractJdbcConnector;
 import com.google.edwmigration.dumper.application.dumper.connector.Connector;
 import com.google.edwmigration.dumper.application.dumper.connector.MetadataConnector;
+import com.google.edwmigration.dumper.application.dumper.connector.ZonedInterval;
+import com.google.edwmigration.dumper.application.dumper.connector.ZonedIntervalIterable;
+import com.google.edwmigration.dumper.application.dumper.connector.ZonedIntervalIterableGenerator;
 import com.google.edwmigration.dumper.application.dumper.handle.Handle;
 import com.google.edwmigration.dumper.application.dumper.handle.JdbcHandle;
 import com.google.edwmigration.dumper.application.dumper.task.DumpMetadataTask;
@@ -41,9 +44,15 @@
 import com.google.edwmigration.dumper.plugin.ext.jdk.annotation.Description;
 import java.sql.Driver;
 import java.time.Clock;
+import java.time.Duration;
+import java.time.ZonedDateTime;
+import java.time.format.DateTimeFormatter;
 import java.util.List;
+import java.util.function.BiFunction;
 import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
 import javax.sql.DataSource;
+import org.apache.commons.lang3.tuple.Pair;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
@@ -65,6 +74,14 @@
 @RespectsArgumentDriverRequired
 @RespectsArgumentUser
 @RespectsArgumentPassword
+@RespectsInput(
+    order = 2000,
+    arg = ConnectorArguments.OPT_START_DATE,
+    description = "Start date for query DAGs data")
+@RespectsInput(
+    order = 2000,
+    arg = ConnectorArguments.OPT_END_DATE,
+    description = "End date for query DAGs data")
 public class AirflowConnector extends AbstractJdbcConnector implements MetadataConnector {
 
   private static final Logger logger = LoggerFactory.getLogger(AirflowConnector.class);
@@ -95,24 +112,96 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = queryRangeFromConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo b/401470428 think about multiple files with this.addTableTaskWithIntervals()
+    addQueryTask(
+        out,
+        "task_instance.csv",
+        "select * from task_instance "
+            + (dateRange != null
+                ? String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight()))
+                : ""));
+
     // Airflow v1.6.0
-    addFullTable(out, "dag_run.csv", "select * from dag_run;");
+    addQueryTask(
+        out,
+        "dag_run.csv",
+        dateRange == null
+            ? "select * from dag_run;"
+            : "select * from dag_run "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
 
     // Airflow v1.10.7
     // analog of DAG's python definition in json
-    addFullTable(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
     // Airflow v1.10.10
-    addFullTable(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+  }
+
+  @Nullable
+  private ZonedIntervalIterable queryRangeFromConnectorArguments(ConnectorArguments arguments) {
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+    if (dateRange == null) {
+      LOG.info("Date ranges was not specified. Generate full table queries.");
+      return null;
+    }
+
+    LOG.info(
+        "Date range for query generation from {} to {} exclusive and increments of one day.",
+        dateRange.getLeft(),
+        dateRange.getRight());
+    return ZonedIntervalIterableGenerator.forDateRangeWithIntervalDuration(
+        dateRange.getLeft(), dateRange.getRight(), Duration.ofDays(1));
   }
 
-  private static void addFullTable(List<? super Task<?>> out, String filename, String sql) {
-    addFullTable(out, filename, sql, TaskCategory.REQUIRED);
+  @Nullable
+  private Pair<ZonedDateTime, ZonedDateTime> dateRange(ConnectorArguments arguments) {
+    if (!(arguments.getStartDate() != null || arguments.getEndDate() != null)) {
+      return null;
+    }
+
+    return Pair.of(arguments.getStartDate(), arguments.getEndDate());
   }
 
-  private static void addFullTable(
+  /**
+   * call example: addTableTaskWithIntervals( out, "task_instance", "select * from task_instance ",
+   * zonedIntervals, (startDate, endDate) -> String.format( " where end_date >= CAST( '%s' as
+   * TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;", dateToSqlFormat(startDate),
+   * dateToSqlFormat(endDate)));
+   */
+  private static void addTableTaskWithIntervals(
+      List<? super Task<?>> out,
+      String filename,
+      String sql,
+      @Nullable ZonedIntervalIterable zonedIntervals,
+      BiFunction<ZonedDateTime, ZonedDateTime, String> toIntervalWhereClause) {
+    if (zonedIntervals == null) {
+      addQueryTask(out, filename, sql, TaskCategory.REQUIRED);
+      return;
+    }
+
+    for (ZonedInterval interval : zonedIntervals) {
+      String calculatedFileName =
+          DateTimeFormatter.ISO_OFFSET_DATE_TIME.format(interval.getStartUTC()) + filename;
+      String calculatedSql =
+          sql + toIntervalWhereClause.apply(interval.getStart(), interval.getEndExclusive());
+      addQueryTask(out, calculatedFileName, calculatedSql);
+    }
+  }
+
+  private static void addQueryTask(List<? super Task<?>> out, String filename, String sql) {
+    addQueryTask(out, filename, sql, TaskCategory.REQUIRED);
+  }
+
+  private static void addQueryTask(
       List<? super Task<?>> out, String filename, String sql, TaskCategory taskCategory) {
     out.add(new JdbcSelectTask(filename, sql, taskCategory));
   }
@@ -144,12 +233,32 @@ public void validate(ConnectorArguments arguments) throws IllegalStateException
       Preconditions.checkState(arguments.getPort() != null, "--port is required with --host");
       Preconditions.checkState(arguments.getSchema() != null, "--schema is required with --host");
     }
+
+    validateDatesRange(arguments);
+  }
+
+  private void validateDatesRange(ConnectorArguments arguments) {
+    ZonedDateTime startDate = arguments.getStartDate();
+    ZonedDateTime endDate = arguments.getEndDate();
+
+    if (startDate != null) {
+      Preconditions.checkNotNull(
+          endDate, "End date must be specified with start date, but was null.");
+      Preconditions.checkState(
+          startDate.isBefore(endDate),
+          "Start date [%s] must be before end date [%s].",
+          startDate,
+          endDate);
+    } else {
+      Preconditions.checkState(
+          endDate == null,
+          "End date can be specified only with start date, but start date was null.");
+    }
   }
 
   @Nonnull
   @Override
   public Handle open(@Nonnull ConnectorArguments arguments) throws Exception {
-
     String jdbcString;
     Driver driver;
 
diff --git a/dumper/app/src/test/java/com/google/edwmigration/dumper/application/dumper/connector/ZonedIntervalIterableTest.java b/dumper/app/src/test/java/com/google/edwmigration/dumper/application/dumper/connector/ZonedIntervalIterableTest.java
index 03817df03..598e3a3e3 100644
--- a/dumper/app/src/test/java/com/google/edwmigration/dumper/application/dumper/connector/ZonedIntervalIterableTest.java
+++ b/dumper/app/src/test/java/com/google/edwmigration/dumper/application/dumper/connector/ZonedIntervalIterableTest.java
@@ -53,7 +53,8 @@ private void testIterable(int expectCount, ZonedIntervalIterable iterable) {
     int actualCount = 0;
     for (ZonedInterval interval : iterable) {
       logger.debug("Interval is {}", interval);
-      assertEquals(iterable.getDuration(), interval.getDuration());
+      assertEquals(iterable.getIntervalDuration(), interval.getDuration());
+
       actualCount++;
     }
     assertEquals(expectCount, actualCount);
diff --git a/dumper/app/src/test/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnectorTest.java b/dumper/app/src/test/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnectorTest.java
index fc4f19dce..027697d64 100644
--- a/dumper/app/src/test/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnectorTest.java
+++ b/dumper/app/src/test/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnectorTest.java
@@ -39,6 +39,52 @@ public class AirflowConnectorTest {
 
   private final AirflowConnector connector = new AirflowConnector();
 
+  private final String validRequiredArgs =
+      "--connector airflow --assessment --driver /home/dir/ --user dbadmin --password"
+          + " --host localhost --port 8080 --schema airflow_db";
+
+  // todo add interval tasks tests
+
+  @Test
+  public void validate_startDateAndEndDate_success() throws Exception {
+    String argsStr = validRequiredArgs + " --start-date=2001-02-20 --end-date=2001-02-25";
+
+    // Act
+    connector.validate(args(argsStr));
+  }
+
+  @Test
+  public void validate_startDateAfterEndDate_throws() {
+    String argsStr = validRequiredArgs + " --start-date=2001-02-20 --end-date=2001-02-20";
+
+    Exception exception =
+        assertThrows(IllegalStateException.class, () -> connector.validate(args(argsStr)));
+    assertEquals(
+        "Start date [2001-02-20T00:00Z] must be before end date [2001-02-20T00:00Z].",
+        exception.getMessage());
+  }
+
+  @Test
+  public void validate_endDateAlone_throws() {
+    String argsStr = validRequiredArgs + " --end-date=2001-02-20";
+
+    Exception exception =
+        assertThrows(IllegalStateException.class, () -> connector.validate(args(argsStr)));
+    assertEquals(
+        "End date can be specified only with start date, but start date was null.",
+        exception.getMessage());
+  }
+
+  @Test
+  public void validate_startDateAlone_throws() {
+    String argsStr = validRequiredArgs + " --start-date=2001-02-20";
+
+    Exception exception =
+        assertThrows(RuntimeException.class, () -> connector.validate(args(argsStr)));
+    assertEquals(
+        "End date must be specified with start date, but was null.", exception.getMessage());
+  }
+
   @Test
   public void validate_databaseParam_isNotSupported() throws Exception {
     String argsStr =
@@ -189,7 +235,7 @@ private void testJdbcSelectTasks(
     List<Task<?>> tasks = new ArrayList<>();
 
     // Act
-    connector.addTasksTo(tasks, null);
+    connector.addTasksTo(tasks, args(validRequiredArgs));
 
     // Assert
     Map<String, String> existingFilesToTables =
@@ -218,7 +264,7 @@ public void addTasksTo_containsRequiredTasks() throws Exception {
     List<Task<?>> tasks = new ArrayList<>();
 
     // Act
-    connector.addTasksTo(tasks, null);
+    connector.addTasksTo(tasks, args(validRequiredArgs));
 
     // Assert
     long dumpMetadataCount = tasks.stream().filter(t -> t instanceof DumpMetadataTask).count();


Reviews:
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-28T13:03:19Z
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-28T13:57:54Z
user: login: kajgol
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-28T14:19:52Z
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-28T14:21:42Z
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-02-28T14:22:16Z
user: login: kajgol
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-02-28T14:30:11Z
user: login: kpiotrowski
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T09:12:40Z
user: login: kpiotrowski
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T09:18:09Z
user: login: kpiotrowski
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T09:20:46Z
user: login: kpiotrowski
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T09:28:28Z
user: login: kpiotrowski
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-03T09:31:16Z
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T13:21:53Z
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T13:23:42Z
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T13:25:24Z
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T13:30:51Z
user: login: kajgol
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-03T13:33:33Z
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T13:37:05Z
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T13:43:49Z
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T13:54:15Z
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T13:58:58Z
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T14:07:46Z
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-03T14:12:33Z
user: login: zaldis
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T09:13:49Z
user: login: zaldis
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T09:17:25Z
user: login: zaldis
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T09:20:50Z
user: login: zaldis
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T09:22:06Z
user: login: zaldis
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T09:29:04Z
user: login: zaldis
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T09:35:24Z
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T09:57:01Z
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T10:00:49Z
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T10:07:27Z
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T10:08:38Z
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T10:09:53Z
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T10:23:17Z
user: login: zaldis
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T10:24:28Z
user: login: zaldis
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-07T10:24:45Z
user: login: zaldis
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T10:28:01Z
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T10:53:41Z
user: login: kpiotrowski
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T10:54:59Z
user: login: kajgol
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-07T11:02:01Z
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T11:26:35Z
user: login: kajgol
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-07T11:37:34Z
user: login: shevek-google
body: My thoughts from discussion:
* Start should temporally be before end.
* Can we make the names sufficiently generic that we can have one set of arguments for query logs, airflow logs, etc, in the future?
* lookback implies with respect to end, so it doesn't "feel" right to say "start" and lookback (because it would be forwards from start). Can we use interval, period, something like that which is agnostic about which endpoint we anchor?

Overall, I appreciate the philosophy that is driving these clarifications, so thank you for asking.
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-24T23:06:44Z
user: login: vladislav-sidorovich
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-25T09:32:47Z

Review comments:
diff_hunk: @@ -95,24 +116,89 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = fromAirflowConnectorArguments(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag ");
+    addTableTaskWithIntervals(
+        out,
+        "task_instance.csv",
+        "select * from task_instance ",
+        zonedIntervals,
+        (startDate, endDate) ->
+            String.format(
+                " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
     // Airflow v1.6.0
-    addFullTable(out, "dag_run.csv", "select * from dag_run;");
+    addTableTaskWithIntervals(
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: vladislav-sidorovich
body: I'm not sure if we should split the query into intervals or just limit the dates. The answer is depends on the data size in DB.
In theory it can be big, not sure if it's a case in practice.
created_at: 2025-02-28T13:03:19Z
author_association: MEMBER
diff_hunk: @@ -95,24 +116,89 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = fromAirflowConnectorArguments(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag ");
+    addTableTaskWithIntervals(
+        out,
+        "task_instance.csv",
+        "select * from task_instance ",
+        zonedIntervals,
+        (startDate, endDate) ->
+            String.format(
+                " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
     // Airflow v1.6.0
-    addFullTable(out, "dag_run.csv", "select * from dag_run;");
+    addTableTaskWithIntervals(
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: vladislav-sidorovich
body: I will keep it in one file for now, because consuming multiple files for metadata may require more work.
created_at: 2025-02-28T13:57:54Z
author_association: MEMBER
diff_hunk: @@ -95,24 +117,123 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = fromAirflowConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "task_instance",
+    //     "select * from task_instance ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+    // // Airflow v1.6.0
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "dag_run",
+    //     "select * from dag_run ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+
+    addQueryTask(
+        out,
+        "task_instance.csv",
+        dateRange == null
+            ? "select * from task_instance;"
+            : "select * from task_instance "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
+
     // Airflow v1.6.0
-    addFullTable(out, "dag_run.csv", "select * from dag_run;");
+    addQueryTask(
+        out,
+        "dag_run.csv",
+        dateRange == null
+            ? "select * from dag_run;"
+            : "select * from dag_run "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
 
     // Airflow v1.10.7
     // analog of DAG's python definition in json
-    addFullTable(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
     // Airflow v1.10.10
-    addFullTable(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+  }
+
+  @Nullable
+  private ZonedIntervalIterable fromAirflowConnectorArguments(ConnectorArguments arguments) {
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+    if (dateRange == null) {
+      LOG.info("Date ranges was not specified. Generate full table queries.");
+      return null;
+    }
+
+    LOG.info(
+        "Date range for query generation from {} to {} exclusive and increments of one day.",
+        dateRange.getLeft(),
+        dateRange.getRight());
+    return ZonedIntervalIterableGenerator.forDateRangeWithIntervalDuration(
+        dateRange.getLeft(), dateRange.getRight(), Duration.ofDays(1));
+  }
+
+  @Nullable
+  private Pair<ZonedDateTime, ZonedDateTime> dateRange(ConnectorArguments arguments) {
+    boolean isDateRangeSpecified =
+        arguments.getStartDate() != null || arguments.getLookbackDays() != null;
+    if (!isDateRangeSpecified) {
+      return null;
+    }
+
+    ZonedDateTime startDate;
+    ZonedDateTime endDate;
+    if (arguments.getLookbackDays() != null) {
+      endDate = arguments.getStartDate(ZonedDateTime.now(ZoneOffset.UTC));
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: kajgol
body: why is the start date used here and not the end date?
created_at: 2025-02-28T14:10:34Z
author_association: COLLABORATOR
diff_hunk: @@ -95,24 +117,123 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = fromAirflowConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "task_instance",
+    //     "select * from task_instance ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+    // // Airflow v1.6.0
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "dag_run",
+    //     "select * from dag_run ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+
+    addQueryTask(
+        out,
+        "task_instance.csv",
+        dateRange == null
+            ? "select * from task_instance;"
+            : "select * from task_instance "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
+
     // Airflow v1.6.0
-    addFullTable(out, "dag_run.csv", "select * from dag_run;");
+    addQueryTask(
+        out,
+        "dag_run.csv",
+        dateRange == null
+            ? "select * from dag_run;"
+            : "select * from dag_run "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
 
     // Airflow v1.10.7
     // analog of DAG's python definition in json
-    addFullTable(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
     // Airflow v1.10.10
-    addFullTable(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+  }
+
+  @Nullable
+  private ZonedIntervalIterable fromAirflowConnectorArguments(ConnectorArguments arguments) {
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: kajgol
body: I think this needs a better name. Starting with "from" suggests that this method creates an AirflowConnector from the arguments.
created_at: 2025-02-28T14:12:47Z
author_association: COLLABORATOR
diff_hunk: @@ -144,12 +265,45 @@ public void validate(ConnectorArguments arguments) throws IllegalStateException
       Preconditions.checkState(arguments.getPort() != null, "--port is required with --host");
       Preconditions.checkState(arguments.getSchema() != null, "--schema is required with --host");
     }
+
+    validateDatesInterval(arguments);
+  }
+
+  private void validateDatesInterval(ConnectorArguments arguments) {
+    // valid inputs:
+    // 1. all null
+    // 2. loockback days
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: kajgol
body: typo
created_at: 2025-02-28T14:15:36Z
author_association: COLLABORATOR
diff_hunk: @@ -144,12 +265,45 @@ public void validate(ConnectorArguments arguments) throws IllegalStateException
       Preconditions.checkState(arguments.getPort() != null, "--port is required with --host");
       Preconditions.checkState(arguments.getSchema() != null, "--schema is required with --host");
     }
+
+    validateDatesInterval(arguments);
+  }
+
+  private void validateDatesInterval(ConnectorArguments arguments) {
+    // valid inputs:
+    // 1. all null
+    // 2. loockback days
+    // 3. loocback with start
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: kajgol
body: shouldn't it be end?
created_at: 2025-02-28T14:16:25Z
author_association: COLLABORATOR
diff_hunk: @@ -144,12 +265,45 @@ public void validate(ConnectorArguments arguments) throws IllegalStateException
       Preconditions.checkState(arguments.getPort() != null, "--port is required with --host");
       Preconditions.checkState(arguments.getSchema() != null, "--schema is required with --host");
     }
+
+    validateDatesInterval(arguments);
+  }
+
+  private void validateDatesInterval(ConnectorArguments arguments) {
+    // valid inputs:
+    // 1. all null
+    // 2. loockback days
+    // 3. loocback with start
+    // 4. start, end
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: kajgol
body: wouldn't a start date also be valid meaning [start;  now) ? I think it could be useful if there was a dump taken on some date and then we want to get new data starting from that date.
created_at: 2025-02-28T14:19:39Z
author_association: COLLABORATOR
diff_hunk: @@ -95,24 +117,123 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = fromAirflowConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "task_instance",
+    //     "select * from task_instance ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+    // // Airflow v1.6.0
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "dag_run",
+    //     "select * from dag_run ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+
+    addQueryTask(
+        out,
+        "task_instance.csv",
+        dateRange == null
+            ? "select * from task_instance;"
+            : "select * from task_instance "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
+
     // Airflow v1.6.0
-    addFullTable(out, "dag_run.csv", "select * from dag_run;");
+    addQueryTask(
+        out,
+        "dag_run.csv",
+        dateRange == null
+            ? "select * from dag_run;"
+            : "select * from dag_run "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
 
     // Airflow v1.10.7
     // analog of DAG's python definition in json
-    addFullTable(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
     // Airflow v1.10.10
-    addFullTable(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+  }
+
+  @Nullable
+  private ZonedIntervalIterable fromAirflowConnectorArguments(ConnectorArguments arguments) {
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: vladislav-sidorovich
body: I'm open for any ideas, let's be consistent with `ZonedIntervalIterableGenerator`.
created_at: 2025-02-28T14:21:42Z
author_association: MEMBER
diff_hunk: @@ -144,12 +265,45 @@ public void validate(ConnectorArguments arguments) throws IllegalStateException
       Preconditions.checkState(arguments.getPort() != null, "--port is required with --host");
       Preconditions.checkState(arguments.getSchema() != null, "--schema is required with --host");
     }
+
+    validateDatesInterval(arguments);
+  }
+
+  private void validateDatesInterval(ConnectorArguments arguments) {
+    // valid inputs:
+    // 1. all null
+    // 2. loockback days
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: vladislav-sidorovich
body: thanks.
created_at: 2025-02-28T14:22:16Z
author_association: MEMBER
diff_hunk: @@ -95,24 +117,123 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = fromAirflowConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "task_instance",
+    //     "select * from task_instance ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+    // // Airflow v1.6.0
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "dag_run",
+    //     "select * from dag_run ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+
+    addQueryTask(
+        out,
+        "task_instance.csv",
+        dateRange == null
+            ? "select * from task_instance;"
+            : "select * from task_instance "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
+
     // Airflow v1.6.0
-    addFullTable(out, "dag_run.csv", "select * from dag_run;");
+    addQueryTask(
+        out,
+        "dag_run.csv",
+        dateRange == null
+            ? "select * from dag_run;"
+            : "select * from dag_run "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
 
     // Airflow v1.10.7
     // analog of DAG's python definition in json
-    addFullTable(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
     // Airflow v1.10.10
-    addFullTable(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+  }
+
+  @Nullable
+  private ZonedIntervalIterable fromAirflowConnectorArguments(ConnectorArguments arguments) {
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: kajgol
body: createQueryRangeFromConnectorArguments? or just queryRangeFromConnectorArguments?
created_at: 2025-02-28T14:30:11Z
author_association: COLLABORATOR
diff_hunk: @@ -229,6 +232,37 @@ public class ConnectorArguments extends DefaultArguments {
           .withOptionalArg()
           .describedAs("sekr1t");
 
+  private final OptionSpec<ZonedDateTime> optionStartDate =
+      parser
+          .accepts(
+              OPT_START_DATE,
+              "Inclusive start date for data to export, value will be truncated to hour")
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/ConnectorArguments.java
user: login: kpiotrowski
body: This description is very generic and suggest that these parameters can be used to limit time range of data exported by the dumper but actually they are used only by the Airflow connector. I suggest to mention that they are valid for airflow connector only or support them in all other connectors (in addition to query-log-*).

Did you think about making the `query-log-*` flags deprecated? The one ones could eventually replace existing ones.
created_at: 2025-03-03T09:12:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -95,24 +117,123 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = fromAirflowConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "task_instance",
+    //     "select * from task_instance ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+    // // Airflow v1.6.0
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "dag_run",
+    //     "select * from dag_run ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+
+    addQueryTask(
+        out,
+        "task_instance.csv",
+        dateRange == null
+            ? "select * from task_instance;"
+            : "select * from task_instance "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
+
     // Airflow v1.6.0
-    addFullTable(out, "dag_run.csv", "select * from dag_run;");
+    addQueryTask(
+        out,
+        "dag_run.csv",
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: kpiotrowski
body: Looks like `dag_run.csv` and `task_instance.csv` use very similar SQL query. How about moving the logic that build query text to a separate function that takes (table_name, dateRange) as arguments to reduce duplicated code?
created_at: 2025-03-03T09:18:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -95,24 +117,123 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = fromAirflowConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "task_instance",
+    //     "select * from task_instance ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+    // // Airflow v1.6.0
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "dag_run",
+    //     "select * from dag_run ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+
+    addQueryTask(
+        out,
+        "task_instance.csv",
+        dateRange == null
+            ? "select * from task_instance;"
+            : "select * from task_instance "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
+
     // Airflow v1.6.0
-    addFullTable(out, "dag_run.csv", "select * from dag_run;");
+    addQueryTask(
+        out,
+        "dag_run.csv",
+        dateRange == null
+            ? "select * from dag_run;"
+            : "select * from dag_run "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
 
     // Airflow v1.10.7
     // analog of DAG's python definition in json
-    addFullTable(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
     // Airflow v1.10.10
-    addFullTable(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+  }
+
+  @Nullable
+  private ZonedIntervalIterable fromAirflowConnectorArguments(ConnectorArguments arguments) {
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+    if (dateRange == null) {
+      LOG.info("Date ranges was not specified. Generate full table queries.");
+      return null;
+    }
+
+    LOG.info(
+        "Date range for query generation from {} to {} exclusive and increments of one day.",
+        dateRange.getLeft(),
+        dateRange.getRight());
+    return ZonedIntervalIterableGenerator.forDateRangeWithIntervalDuration(
+        dateRange.getLeft(), dateRange.getRight(), Duration.ofDays(1));
+  }
+
+  @Nullable
+  private Pair<ZonedDateTime, ZonedDateTime> dateRange(ConnectorArguments arguments) {
+    boolean isDateRangeSpecified =
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: kpiotrowski
body: nit: I think `isDateRangeSpecified` is redundant. You can immediately return if (arguments.getStartDate() == null || arguments.getLookbackDays() == null)
created_at: 2025-03-03T09:20:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -95,24 +117,123 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = fromAirflowConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files
+    // addTableTaskWithIntervals(
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: kpiotrowski
body: nit: I'd  avoid keeping unused and commented code in the repository. If you will be working on this later I'd suggest moving it to the separate branch/PR.
created_at: 2025-03-03T09:28:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -144,12 +265,45 @@ public void validate(ConnectorArguments arguments) throws IllegalStateException
       Preconditions.checkState(arguments.getPort() != null, "--port is required with --host");
       Preconditions.checkState(arguments.getSchema() != null, "--schema is required with --host");
     }
+
+    validateDatesInterval(arguments);
+  }
+
+  private void validateDatesInterval(ConnectorArguments arguments) {
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: kpiotrowski
body: Shouldn't the validation be executed on higher level than the airflow connector (or at least function moved to some shared space)? New flags are not defined in the scope of the airflow connector and they could be used by other connectors too.
created_at: 2025-03-03T09:31:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -95,24 +117,123 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = fromAirflowConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "task_instance",
+    //     "select * from task_instance ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+    // // Airflow v1.6.0
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "dag_run",
+    //     "select * from dag_run ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+
+    addQueryTask(
+        out,
+        "task_instance.csv",
+        dateRange == null
+            ? "select * from task_instance;"
+            : "select * from task_instance "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
+
     // Airflow v1.6.0
-    addFullTable(out, "dag_run.csv", "select * from dag_run;");
+    addQueryTask(
+        out,
+        "dag_run.csv",
+        dateRange == null
+            ? "select * from dag_run;"
+            : "select * from dag_run "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
 
     // Airflow v1.10.7
     // analog of DAG's python definition in json
-    addFullTable(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
     // Airflow v1.10.10
-    addFullTable(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+  }
+
+  @Nullable
+  private ZonedIntervalIterable fromAirflowConnectorArguments(ConnectorArguments arguments) {
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+    if (dateRange == null) {
+      LOG.info("Date ranges was not specified. Generate full table queries.");
+      return null;
+    }
+
+    LOG.info(
+        "Date range for query generation from {} to {} exclusive and increments of one day.",
+        dateRange.getLeft(),
+        dateRange.getRight());
+    return ZonedIntervalIterableGenerator.forDateRangeWithIntervalDuration(
+        dateRange.getLeft(), dateRange.getRight(), Duration.ofDays(1));
+  }
+
+  @Nullable
+  private Pair<ZonedDateTime, ZonedDateTime> dateRange(ConnectorArguments arguments) {
+    boolean isDateRangeSpecified =
+        arguments.getStartDate() != null || arguments.getLookbackDays() != null;
+    if (!isDateRangeSpecified) {
+      return null;
+    }
+
+    ZonedDateTime startDate;
+    ZonedDateTime endDate;
+    if (arguments.getLookbackDays() != null) {
+      endDate = arguments.getStartDate(ZonedDateTime.now(ZoneOffset.UTC));
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: vladislav-sidorovich
body: It's combined with `lookbackDays` not with `lookforward*`. I have unit-test to cover this logic `validate_startDateAndLookbackDays_success`, feel free to check.
created_at: 2025-03-03T13:21:53Z
author_association: MEMBER
diff_hunk: @@ -95,24 +117,123 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = fromAirflowConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "task_instance",
+    //     "select * from task_instance ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+    // // Airflow v1.6.0
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "dag_run",
+    //     "select * from dag_run ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+
+    addQueryTask(
+        out,
+        "task_instance.csv",
+        dateRange == null
+            ? "select * from task_instance;"
+            : "select * from task_instance "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
+
     // Airflow v1.6.0
-    addFullTable(out, "dag_run.csv", "select * from dag_run;");
+    addQueryTask(
+        out,
+        "dag_run.csv",
+        dateRange == null
+            ? "select * from dag_run;"
+            : "select * from dag_run "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
 
     // Airflow v1.10.7
     // analog of DAG's python definition in json
-    addFullTable(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
     // Airflow v1.10.10
-    addFullTable(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+  }
+
+  @Nullable
+  private ZonedIntervalIterable fromAirflowConnectorArguments(ConnectorArguments arguments) {
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: vladislav-sidorovich
body: sounds good, done.
created_at: 2025-03-03T13:23:42Z
author_association: MEMBER
diff_hunk: @@ -144,12 +265,45 @@ public void validate(ConnectorArguments arguments) throws IllegalStateException
       Preconditions.checkState(arguments.getPort() != null, "--port is required with --host");
       Preconditions.checkState(arguments.getSchema() != null, "--schema is required with --host");
     }
+
+    validateDatesInterval(arguments);
+  }
+
+  private void validateDatesInterval(ConnectorArguments arguments) {
+    // valid inputs:
+    // 1. all null
+    // 2. loockback days
+    // 3. loocback with start
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: vladislav-sidorovich
body: nope, it's start. There is unit-test for that and the description in this PR as well.
created_at: 2025-03-03T13:25:24Z
author_association: MEMBER
diff_hunk: @@ -144,12 +265,45 @@ public void validate(ConnectorArguments arguments) throws IllegalStateException
       Preconditions.checkState(arguments.getPort() != null, "--port is required with --host");
       Preconditions.checkState(arguments.getSchema() != null, "--schema is required with --host");
     }
+
+    validateDatesInterval(arguments);
+  }
+
+  private void validateDatesInterval(ConnectorArguments arguments) {
+    // valid inputs:
+    // 1. all null
+    // 2. loockback days
+    // 3. loocback with start
+    // 4. start, end
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: vladislav-sidorovich
body: Well, this is actually good point which I didn't think about initially. 

after a little thought, I come to conclusion that such usage may be useful, but it might lead to the mistakes of API usage (user just forgot a parameter) as well. So I prefer to keep more strict combination of parameters here.  
created_at: 2025-03-03T13:30:51Z
author_association: MEMBER
diff_hunk: @@ -95,24 +117,123 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = fromAirflowConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "task_instance",
+    //     "select * from task_instance ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+    // // Airflow v1.6.0
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "dag_run",
+    //     "select * from dag_run ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+
+    addQueryTask(
+        out,
+        "task_instance.csv",
+        dateRange == null
+            ? "select * from task_instance;"
+            : "select * from task_instance "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
+
     // Airflow v1.6.0
-    addFullTable(out, "dag_run.csv", "select * from dag_run;");
+    addQueryTask(
+        out,
+        "dag_run.csv",
+        dateRange == null
+            ? "select * from dag_run;"
+            : "select * from dag_run "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
 
     // Airflow v1.10.7
     // analog of DAG's python definition in json
-    addFullTable(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
     // Airflow v1.10.10
-    addFullTable(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+  }
+
+  @Nullable
+  private ZonedIntervalIterable fromAirflowConnectorArguments(ConnectorArguments arguments) {
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+    if (dateRange == null) {
+      LOG.info("Date ranges was not specified. Generate full table queries.");
+      return null;
+    }
+
+    LOG.info(
+        "Date range for query generation from {} to {} exclusive and increments of one day.",
+        dateRange.getLeft(),
+        dateRange.getRight());
+    return ZonedIntervalIterableGenerator.forDateRangeWithIntervalDuration(
+        dateRange.getLeft(), dateRange.getRight(), Duration.ofDays(1));
+  }
+
+  @Nullable
+  private Pair<ZonedDateTime, ZonedDateTime> dateRange(ConnectorArguments arguments) {
+    boolean isDateRangeSpecified =
+        arguments.getStartDate() != null || arguments.getLookbackDays() != null;
+    if (!isDateRangeSpecified) {
+      return null;
+    }
+
+    ZonedDateTime startDate;
+    ZonedDateTime endDate;
+    if (arguments.getLookbackDays() != null) {
+      endDate = arguments.getStartDate(ZonedDateTime.now(ZoneOffset.UTC));
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: kajgol
body: It is but it's inconsistent since the interval is sometimes defined as [start, end) and sometimes [start -x, start) so "start" can mean the start or end of the interval.
created_at: 2025-03-03T13:33:33Z
author_association: COLLABORATOR
diff_hunk: @@ -229,6 +232,37 @@ public class ConnectorArguments extends DefaultArguments {
           .withOptionalArg()
           .describedAs("sekr1t");
 
+  private final OptionSpec<ZonedDateTime> optionStartDate =
+      parser
+          .accepts(
+              OPT_START_DATE,
+              "Inclusive start date for data to export, value will be truncated to hour")
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/ConnectorArguments.java
user: login: vladislav-sidorovich
body: 1. Yes, the description is generic because the parameters are generic too. 
2. You are right, currently only Airflow connector supports these parameters. However not all parameters are supported by all connectors, so I think it's ok. In my opinion the usage will be extended eventually. 
3. I did think about  deprecating `query-log-*`. I think it's too early do it right now. In general I like that we have `query-logs` connectors and and `query-log-*` parameters, it's more specific, but of courser the `query-log-*` params maybe replaced with these. I think we should leave with both parameters for some time and later with more cases make a decision. 

created_at: 2025-03-03T13:37:05Z
author_association: MEMBER
diff_hunk: @@ -95,24 +117,123 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = fromAirflowConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "task_instance",
+    //     "select * from task_instance ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+    // // Airflow v1.6.0
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "dag_run",
+    //     "select * from dag_run ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+
+    addQueryTask(
+        out,
+        "task_instance.csv",
+        dateRange == null
+            ? "select * from task_instance;"
+            : "select * from task_instance "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
+
     // Airflow v1.6.0
-    addFullTable(out, "dag_run.csv", "select * from dag_run;");
+    addQueryTask(
+        out,
+        "dag_run.csv",
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: vladislav-sidorovich
body: I would say that all the sql queries are similar here, however I would not extract a new function for that reason. 

Downside of removing "duplication" is introduction coupling between the components. In the current case those tables are independent and queries may change in the future independently, so we do not need to couple the components.   

The question: `Will these queries change at the same time for the same reason in the future?`
The answer: `No, the tables are independent.`
created_at: 2025-03-03T13:43:49Z
author_association: MEMBER
diff_hunk: @@ -95,24 +117,123 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = fromAirflowConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files
+    // addTableTaskWithIntervals(
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: vladislav-sidorovich
body: Well, it's just and example of API call, but I agree it's `dead code`.

Ok, let's move it to a bit different comment `/**`, so now it's a javadoc 😎 and not a `dead code` anymore.
created_at: 2025-03-03T13:54:15Z
author_association: MEMBER
diff_hunk: @@ -144,12 +265,45 @@ public void validate(ConnectorArguments arguments) throws IllegalStateException
       Preconditions.checkState(arguments.getPort() != null, "--port is required with --host");
       Preconditions.checkState(arguments.getSchema() != null, "--schema is required with --host");
     }
+
+    validateDatesInterval(arguments);
+  }
+
+  private void validateDatesInterval(ConnectorArguments arguments) {
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: vladislav-sidorovich
body: Yes, I think this should eventually be extracted into some new abstraction. 
However, it is better to do this when adopting the 2nd or 3rd connector, since this will be the state when the value appears and the use case will be clearer.
created_at: 2025-03-03T13:58:58Z
author_association: MEMBER
diff_hunk: @@ -95,24 +117,123 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = fromAirflowConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "task_instance",
+    //     "select * from task_instance ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+    // // Airflow v1.6.0
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "dag_run",
+    //     "select * from dag_run ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+
+    addQueryTask(
+        out,
+        "task_instance.csv",
+        dateRange == null
+            ? "select * from task_instance;"
+            : "select * from task_instance "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
+
     // Airflow v1.6.0
-    addFullTable(out, "dag_run.csv", "select * from dag_run;");
+    addQueryTask(
+        out,
+        "dag_run.csv",
+        dateRange == null
+            ? "select * from dag_run;"
+            : "select * from dag_run "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
 
     // Airflow v1.10.7
     // analog of DAG's python definition in json
-    addFullTable(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
     // Airflow v1.10.10
-    addFullTable(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+  }
+
+  @Nullable
+  private ZonedIntervalIterable fromAirflowConnectorArguments(ConnectorArguments arguments) {
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+    if (dateRange == null) {
+      LOG.info("Date ranges was not specified. Generate full table queries.");
+      return null;
+    }
+
+    LOG.info(
+        "Date range for query generation from {} to {} exclusive and increments of one day.",
+        dateRange.getLeft(),
+        dateRange.getRight());
+    return ZonedIntervalIterableGenerator.forDateRangeWithIntervalDuration(
+        dateRange.getLeft(), dateRange.getRight(), Duration.ofDays(1));
+  }
+
+  @Nullable
+  private Pair<ZonedDateTime, ZonedDateTime> dateRange(ConnectorArguments arguments) {
+    boolean isDateRangeSpecified =
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: vladislav-sidorovich
body: That right, but there is a technic of self documented code. The variable is introduced to describe the boolean expression instead of writing a comment. 
created_at: 2025-03-03T14:07:46Z
author_association: MEMBER
diff_hunk: @@ -95,24 +117,123 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = fromAirflowConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "task_instance",
+    //     "select * from task_instance ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+    // // Airflow v1.6.0
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "dag_run",
+    //     "select * from dag_run ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+
+    addQueryTask(
+        out,
+        "task_instance.csv",
+        dateRange == null
+            ? "select * from task_instance;"
+            : "select * from task_instance "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
+
     // Airflow v1.6.0
-    addFullTable(out, "dag_run.csv", "select * from dag_run;");
+    addQueryTask(
+        out,
+        "dag_run.csv",
+        dateRange == null
+            ? "select * from dag_run;"
+            : "select * from dag_run "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
 
     // Airflow v1.10.7
     // analog of DAG's python definition in json
-    addFullTable(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
     // Airflow v1.10.10
-    addFullTable(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+  }
+
+  @Nullable
+  private ZonedIntervalIterable fromAirflowConnectorArguments(ConnectorArguments arguments) {
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+    if (dateRange == null) {
+      LOG.info("Date ranges was not specified. Generate full table queries.");
+      return null;
+    }
+
+    LOG.info(
+        "Date range for query generation from {} to {} exclusive and increments of one day.",
+        dateRange.getLeft(),
+        dateRange.getRight());
+    return ZonedIntervalIterableGenerator.forDateRangeWithIntervalDuration(
+        dateRange.getLeft(), dateRange.getRight(), Duration.ofDays(1));
+  }
+
+  @Nullable
+  private Pair<ZonedDateTime, ZonedDateTime> dateRange(ConnectorArguments arguments) {
+    boolean isDateRangeSpecified =
+        arguments.getStartDate() != null || arguments.getLookbackDays() != null;
+    if (!isDateRangeSpecified) {
+      return null;
+    }
+
+    ZonedDateTime startDate;
+    ZonedDateTime endDate;
+    if (arguments.getLookbackDays() != null) {
+      endDate = arguments.getStartDate(ZonedDateTime.now(ZoneOffset.UTC));
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: vladislav-sidorovich
body: that's true. However `start` alway used with an additional parameter:
1. start + end
2. start + lookback

So, the combination of the parameters doesn't look misleading to me. But let's keep the conversation open to collect more opinions if any.    
created_at: 2025-03-03T14:12:33Z
author_association: MEMBER
diff_hunk: @@ -229,6 +232,37 @@ public class ConnectorArguments extends DefaultArguments {
           .withOptionalArg()
           .describedAs("sekr1t");
 
+  private final OptionSpec<ZonedDateTime> optionStartDate =
+      parser
+          .accepts(
+              OPT_START_DATE,
+              "Inclusive start date for data to export, value will be truncated to hour")
+          .withOptionalArg()
+          .ofType(Date.class)
+          .withValuesConvertedBy(ZonedParser.withDefaultPattern(DayOffset.START_OF_DAY))
+          .describedAs("2001-01-15[ 00:00:00.[000]]");
+
+  private final OptionSpec<ZonedDateTime> optionEndDate =
+      parser
+          .accepts(
+              OPT_END_DATE,
+              "Inclusive end date for data to export, value will be truncated to hour")
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/ConnectorArguments.java
user: login: zaldis
body: ```suggestion
              "Exclusive end date for data to export, value will be truncated to hour")
```
Based on PR description this parameter should not be inclusive.
created_at: 2025-03-07T09:13:49Z
author_association: MEMBER
diff_hunk: @@ -229,6 +232,37 @@ public class ConnectorArguments extends DefaultArguments {
           .withOptionalArg()
           .describedAs("sekr1t");
 
+  private final OptionSpec<ZonedDateTime> optionStartDate =
+      parser
+          .accepts(
+              OPT_START_DATE,
+              "Inclusive start date for data to export, value will be truncated to hour")
+          .withOptionalArg()
+          .ofType(Date.class)
+          .withValuesConvertedBy(ZonedParser.withDefaultPattern(DayOffset.START_OF_DAY))
+          .describedAs("2001-01-15[ 00:00:00.[000]]");
+
+  private final OptionSpec<ZonedDateTime> optionEndDate =
+      parser
+          .accepts(
+              OPT_END_DATE,
+              "Inclusive end date for data to export, value will be truncated to hour")
+          .withOptionalArg()
+          .ofType(Date.class)
+          .withValuesConvertedBy(ZonedParser.withDefaultPattern(DayOffset.START_OF_DAY))
+          .describedAs("2001-01-15[ 00:00:00.[000]]");
+
+  private final OptionSpec<Integer> optionLookbackDays =
+      parser
+          .accepts(
+              OPT_LOOKBACK_DAYS,
+              "The N days back of data to export from current moment. "
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/ConnectorArguments.java
user: login: zaldis
body: Please, add more details about `start_date` parameter (it's not always a current moment).
created_at: 2025-03-07T09:17:25Z
author_association: MEMBER
diff_hunk: @@ -35,5 +35,5 @@
     description = RespectsArgumentQueryLogDays.DESCRIPTION)
 public @interface RespectsArgumentQueryLogDays {
 
-  public static final String DESCRIPTION = "The number of days of query history to dump.";
+  String DESCRIPTION = "The number of days of query history to dump.";
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/annotations/RespectsArgumentQueryLogDays.java
user: login: zaldis
body: Shouldn't it be with `final` modifier?
```suggestion
  final String DESCRIPTION = "The number of days of query history to dump.";
```
created_at: 2025-03-07T09:20:49Z
author_association: MEMBER
diff_hunk: @@ -35,5 +35,5 @@
     description = RespectsArgumentQueryLogDays.DESCRIPTION)
 public @interface RespectsArgumentQueryLogDays {
 
-  public static final String DESCRIPTION = "The number of days of query history to dump.";
+  String DESCRIPTION = "The number of days of query history to dump.";
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/annotations/RespectsArgumentQueryLogDays.java
user: login: zaldis
body: Or looks like it's final by default
created_at: 2025-03-07T09:22:06Z
author_association: MEMBER
diff_hunk: @@ -95,24 +117,107 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = queryRangeFromConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files with this.addTableTaskWithIntervals()
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: zaldis
body: Any bug ticket for this?
created_at: 2025-03-07T09:29:04Z
author_association: MEMBER
diff_hunk: @@ -95,24 +117,107 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = queryRangeFromConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files with this.addTableTaskWithIntervals()
+    addQueryTask(
+        out,
+        "task_instance.csv",
+        dateRange == null
+            ? "select * from task_instance;"
+            : "select * from task_instance "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: zaldis
body: If pre-build the filter query separately it might be easier to read and debug.
```suggestion
    String queryFilter = "";
    if (dateRange != null) {
      queryFilter = String.format(
           "where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
           dateToSqlFormat(dateRange.getLeft()),
           dateToSqlFormat(dateRange.getRight())
       )
    }
    addQueryTask(
        out,
        "task_instance.csv",
        String.format("select * from task_instance %s;", queryFilter)
```
created_at: 2025-03-07T09:35:24Z
author_association: MEMBER
diff_hunk: @@ -35,5 +35,5 @@
     description = RespectsArgumentQueryLogDays.DESCRIPTION)
 public @interface RespectsArgumentQueryLogDays {
 
-  public static final String DESCRIPTION = "The number of days of query history to dump.";
+  String DESCRIPTION = "The number of days of query history to dump.";
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/annotations/RespectsArgumentQueryLogDays.java
user: login: vladislav-sidorovich
body: yes
created_at: 2025-03-07T09:57:01Z
author_association: MEMBER
diff_hunk: @@ -229,6 +232,37 @@ public class ConnectorArguments extends DefaultArguments {
           .withOptionalArg()
           .describedAs("sekr1t");
 
+  private final OptionSpec<ZonedDateTime> optionStartDate =
+      parser
+          .accepts(
+              OPT_START_DATE,
+              "Inclusive start date for data to export, value will be truncated to hour")
+          .withOptionalArg()
+          .ofType(Date.class)
+          .withValuesConvertedBy(ZonedParser.withDefaultPattern(DayOffset.START_OF_DAY))
+          .describedAs("2001-01-15[ 00:00:00.[000]]");
+
+  private final OptionSpec<ZonedDateTime> optionEndDate =
+      parser
+          .accepts(
+              OPT_END_DATE,
+              "Inclusive end date for data to export, value will be truncated to hour")
+          .withOptionalArg()
+          .ofType(Date.class)
+          .withValuesConvertedBy(ZonedParser.withDefaultPattern(DayOffset.START_OF_DAY))
+          .describedAs("2001-01-15[ 00:00:00.[000]]");
+
+  private final OptionSpec<Integer> optionLookbackDays =
+      parser
+          .accepts(
+              OPT_LOOKBACK_DAYS,
+              "The N days back of data to export from current moment. "
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/ConnectorArguments.java
user: login: vladislav-sidorovich
body: There is next line:
```
+ "Can be combined with --"
                  + OPT_START_DATE)
```
created_at: 2025-03-07T10:00:49Z
author_association: MEMBER
diff_hunk: @@ -229,6 +232,37 @@ public class ConnectorArguments extends DefaultArguments {
           .withOptionalArg()
           .describedAs("sekr1t");
 
+  private final OptionSpec<ZonedDateTime> optionStartDate =
+      parser
+          .accepts(
+              OPT_START_DATE,
+              "Inclusive start date for data to export, value will be truncated to hour")
+          .withOptionalArg()
+          .ofType(Date.class)
+          .withValuesConvertedBy(ZonedParser.withDefaultPattern(DayOffset.START_OF_DAY))
+          .describedAs("2001-01-15[ 00:00:00.[000]]");
+
+  private final OptionSpec<ZonedDateTime> optionEndDate =
+      parser
+          .accepts(
+              OPT_END_DATE,
+              "Inclusive end date for data to export, value will be truncated to hour")
+          .withOptionalArg()
+          .ofType(Date.class)
+          .withValuesConvertedBy(ZonedParser.withDefaultPattern(DayOffset.START_OF_DAY))
+          .describedAs("2001-01-15[ 00:00:00.[000]]");
+
+  private final OptionSpec<Integer> optionLookbackDays =
+      parser
+          .accepts(
+              OPT_LOOKBACK_DAYS,
+              "The N days back of data to export from current moment. "
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/ConnectorArguments.java
user: login: vladislav-sidorovich
body: the description updated to be more clear
created_at: 2025-03-07T10:07:27Z
author_association: MEMBER
diff_hunk: @@ -229,6 +232,37 @@ public class ConnectorArguments extends DefaultArguments {
           .withOptionalArg()
           .describedAs("sekr1t");
 
+  private final OptionSpec<ZonedDateTime> optionStartDate =
+      parser
+          .accepts(
+              OPT_START_DATE,
+              "Inclusive start date for data to export, value will be truncated to hour")
+          .withOptionalArg()
+          .ofType(Date.class)
+          .withValuesConvertedBy(ZonedParser.withDefaultPattern(DayOffset.START_OF_DAY))
+          .describedAs("2001-01-15[ 00:00:00.[000]]");
+
+  private final OptionSpec<ZonedDateTime> optionEndDate =
+      parser
+          .accepts(
+              OPT_END_DATE,
+              "Inclusive end date for data to export, value will be truncated to hour")
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/ConnectorArguments.java
user: login: vladislav-sidorovich
body: Yes, thank you. fixed.
created_at: 2025-03-07T10:08:37Z
author_association: MEMBER
diff_hunk: @@ -95,24 +117,107 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = queryRangeFromConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files with this.addTableTaskWithIntervals()
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: vladislav-sidorovich
body: I would say it should usage experience and not the ticket. once we will have some feedbacks, we will "think about it" 
created_at: 2025-03-07T10:09:53Z
author_association: MEMBER
diff_hunk: @@ -95,24 +117,107 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = queryRangeFromConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files with this.addTableTaskWithIntervals()
+    addQueryTask(
+        out,
+        "task_instance.csv",
+        dateRange == null
+            ? "select * from task_instance;"
+            : "select * from task_instance "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: vladislav-sidorovich
body: ```
String filter =
        dateRange != null
            ? String.format(
                " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
                dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight()))
            : "";
addQueryTask(out, "task_instance.csv", "select * from task_instance " + filter);
```
vs
```
    addQueryTask(
        out,
        "task_instance.csv",
        "select * from task_instance "
            + (dateRange != null
                ? String.format(
                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight()))
                : ""));

```
vs    
```
  addQueryTask(
        out,
        "task_instance.csv",
        dateRange == null
            ? "select * from task_instance;"
            : "select * from task_instance "
                + String.format(
                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
```

Not a big difference from my point of view. The variable looks good, but it will require to introduce such variable for  several method or use code blocks `{ ... }`.  

So, I will keep it as is.
created_at: 2025-03-07T10:23:17Z
author_association: MEMBER
diff_hunk: @@ -95,24 +117,107 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = queryRangeFromConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files with this.addTableTaskWithIntervals()
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: zaldis
body: We can track it in the ticket for visibility and close it if it will not be relevant in the future.
created_at: 2025-03-07T10:24:28Z
author_association: MEMBER
diff_hunk: @@ -95,24 +117,107 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = queryRangeFromConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files with this.addTableTaskWithIntervals()
+    addQueryTask(
+        out,
+        "task_instance.csv",
+        dateRange == null
+            ? "select * from task_instance;"
+            : "select * from task_instance "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: zaldis
body: The main difference that you can log your filter or set a breakpoint for debugging purposes (right after your filter string will be generated). With current implementation you have to do it in `addQueryTask` which is called multiple times (from different places). Hence it will be harder to identify required call.
created_at: 2025-03-07T10:28:01Z
author_association: MEMBER
diff_hunk: @@ -95,24 +117,107 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = queryRangeFromConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files with this.addTableTaskWithIntervals()
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: vladislav-sidorovich
body: added
created_at: 2025-03-07T10:53:41Z
author_association: MEMBER
diff_hunk: @@ -95,24 +116,89 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = fromAirflowConnectorArguments(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag ");
+    addTableTaskWithIntervals(
+        out,
+        "task_instance.csv",
+        "select * from task_instance ",
+        zonedIntervals,
+        (startDate, endDate) ->
+            String.format(
+                " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
     // Airflow v1.6.0
-    addFullTable(out, "dag_run.csv", "select * from dag_run;");
+    addTableTaskWithIntervals(
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: kpiotrowski
body: If there will be a need to split it into multiple smaller, interval based files I'd consider ingesting it as a time-series data. We have a process for it. Airflow connector could be use by both metadata and time-series loaders.
created_at: 2025-03-07T10:54:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -95,24 +117,123 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = fromAirflowConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "task_instance",
+    //     "select * from task_instance ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+    // // Airflow v1.6.0
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "dag_run",
+    //     "select * from dag_run ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+
+    addQueryTask(
+        out,
+        "task_instance.csv",
+        dateRange == null
+            ? "select * from task_instance;"
+            : "select * from task_instance "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
+
     // Airflow v1.6.0
-    addFullTable(out, "dag_run.csv", "select * from dag_run;");
+    addQueryTask(
+        out,
+        "dag_run.csv",
+        dateRange == null
+            ? "select * from dag_run;"
+            : "select * from dag_run "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
 
     // Airflow v1.10.7
     // analog of DAG's python definition in json
-    addFullTable(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
     // Airflow v1.10.10
-    addFullTable(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+  }
+
+  @Nullable
+  private ZonedIntervalIterable fromAirflowConnectorArguments(ConnectorArguments arguments) {
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+    if (dateRange == null) {
+      LOG.info("Date ranges was not specified. Generate full table queries.");
+      return null;
+    }
+
+    LOG.info(
+        "Date range for query generation from {} to {} exclusive and increments of one day.",
+        dateRange.getLeft(),
+        dateRange.getRight());
+    return ZonedIntervalIterableGenerator.forDateRangeWithIntervalDuration(
+        dateRange.getLeft(), dateRange.getRight(), Duration.ofDays(1));
+  }
+
+  @Nullable
+  private Pair<ZonedDateTime, ZonedDateTime> dateRange(ConnectorArguments arguments) {
+    boolean isDateRangeSpecified =
+        arguments.getStartDate() != null || arguments.getLookbackDays() != null;
+    if (!isDateRangeSpecified) {
+      return null;
+    }
+
+    ZonedDateTime startDate;
+    ZonedDateTime endDate;
+    if (arguments.getLookbackDays() != null) {
+      endDate = arguments.getStartDate(ZonedDateTime.now(ZoneOffset.UTC));
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: kajgol
body: I'm starting to wonder if we need this case at all. I can see a usecase for --lookback=7 days from now, or --start-date=last tuesday because that's when I run the previous dump, or --start-date=last tuesday and --end-date=last-saturday because I want a specific range, but why would I want something like 7 days before last tuesday?
created_at: 2025-03-07T11:02:00Z
author_association: COLLABORATOR
diff_hunk: @@ -95,24 +117,123 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = fromAirflowConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "task_instance",
+    //     "select * from task_instance ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+    // // Airflow v1.6.0
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "dag_run",
+    //     "select * from dag_run ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+
+    addQueryTask(
+        out,
+        "task_instance.csv",
+        dateRange == null
+            ? "select * from task_instance;"
+            : "select * from task_instance "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
+
     // Airflow v1.6.0
-    addFullTable(out, "dag_run.csv", "select * from dag_run;");
+    addQueryTask(
+        out,
+        "dag_run.csv",
+        dateRange == null
+            ? "select * from dag_run;"
+            : "select * from dag_run "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
 
     // Airflow v1.10.7
     // analog of DAG's python definition in json
-    addFullTable(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
     // Airflow v1.10.10
-    addFullTable(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+  }
+
+  @Nullable
+  private ZonedIntervalIterable fromAirflowConnectorArguments(ConnectorArguments arguments) {
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+    if (dateRange == null) {
+      LOG.info("Date ranges was not specified. Generate full table queries.");
+      return null;
+    }
+
+    LOG.info(
+        "Date range for query generation from {} to {} exclusive and increments of one day.",
+        dateRange.getLeft(),
+        dateRange.getRight());
+    return ZonedIntervalIterableGenerator.forDateRangeWithIntervalDuration(
+        dateRange.getLeft(), dateRange.getRight(), Duration.ofDays(1));
+  }
+
+  @Nullable
+  private Pair<ZonedDateTime, ZonedDateTime> dateRange(ConnectorArguments arguments) {
+    boolean isDateRangeSpecified =
+        arguments.getStartDate() != null || arguments.getLookbackDays() != null;
+    if (!isDateRangeSpecified) {
+      return null;
+    }
+
+    ZonedDateTime startDate;
+    ZonedDateTime endDate;
+    if (arguments.getLookbackDays() != null) {
+      endDate = arguments.getStartDate(ZonedDateTime.now(ZoneOffset.UTC));
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: vladislav-sidorovich
body: 1. Can be connector used without this case? - yes, start/end is enough to cover all ranges.
2. is there additional benefit for such use case? - yes, it's simpler way to specify the range.
3. is the benefit significant? - it's hard to say/measure. (later on we will be able to measure)

> --start-date=last tuesday because that's when I run the previous dump

But you actually raised an interesting use case. What if I will run the dumper 3 times and I want cover the ranges [t1 - t2 - t3 - t4]? It doesn't look like a use case, we just need to run dumper once for [t1 - t4] where t4 can be now. 
We can go forward and think about an `incremental assessment`, run dumper each day/week and append data to the existing assessment... It sounds interesting, but it's not the case for now. 

created_at: 2025-03-07T11:26:34Z
author_association: MEMBER
diff_hunk: @@ -95,24 +117,123 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = fromAirflowConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "task_instance",
+    //     "select * from task_instance ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+    // // Airflow v1.6.0
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "dag_run",
+    //     "select * from dag_run ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+
+    addQueryTask(
+        out,
+        "task_instance.csv",
+        dateRange == null
+            ? "select * from task_instance;"
+            : "select * from task_instance "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
+
     // Airflow v1.6.0
-    addFullTable(out, "dag_run.csv", "select * from dag_run;");
+    addQueryTask(
+        out,
+        "dag_run.csv",
+        dateRange == null
+            ? "select * from dag_run;"
+            : "select * from dag_run "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
 
     // Airflow v1.10.7
     // analog of DAG's python definition in json
-    addFullTable(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
     // Airflow v1.10.10
-    addFullTable(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+  }
+
+  @Nullable
+  private ZonedIntervalIterable fromAirflowConnectorArguments(ConnectorArguments arguments) {
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+    if (dateRange == null) {
+      LOG.info("Date ranges was not specified. Generate full table queries.");
+      return null;
+    }
+
+    LOG.info(
+        "Date range for query generation from {} to {} exclusive and increments of one day.",
+        dateRange.getLeft(),
+        dateRange.getRight());
+    return ZonedIntervalIterableGenerator.forDateRangeWithIntervalDuration(
+        dateRange.getLeft(), dateRange.getRight(), Duration.ofDays(1));
+  }
+
+  @Nullable
+  private Pair<ZonedDateTime, ZonedDateTime> dateRange(ConnectorArguments arguments) {
+    boolean isDateRangeSpecified =
+        arguments.getStartDate() != null || arguments.getLookbackDays() != null;
+    if (!isDateRangeSpecified) {
+      return null;
+    }
+
+    ZonedDateTime startDate;
+    ZonedDateTime endDate;
+    if (arguments.getLookbackDays() != null) {
+      endDate = arguments.getStartDate(ZonedDateTime.now(ZoneOffset.UTC));
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: kajgol
body: "is there additional benefit for such use case? - yes, it's simpler way to specify the range."
It's not simpler if it's confusing ;-)
created_at: 2025-03-07T11:37:33Z
author_association: COLLABORATOR
diff_hunk: @@ -95,24 +117,123 @@ public void addTasksTo(@Nonnull List<? super Task<?>> out, @Nonnull ConnectorArg
     out.add(new DumpMetadataTask(arguments, FORMAT_NAME));
     out.add(new FormatTask(FORMAT_NAME));
 
+    ZonedIntervalIterable zonedIntervals = fromAirflowConnectorArguments(arguments);
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+
     // Airflow v1.5.0
-    addFullTable(out, "dag.csv", "select * from dag;");
-    addFullTable(out, "task_instance.csv", "select * from task_instance;");
+    addQueryTask(out, "dag.csv", "select * from dag;");
+    // todo think about multiple files
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "task_instance",
+    //     "select * from task_instance ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+    // // Airflow v1.6.0
+    // addTableTaskWithIntervals(
+    //     out,
+    //     "dag_run",
+    //     "select * from dag_run ",
+    //     zonedIntervals,
+    //     (startDate, endDate) ->
+    //         String.format(
+    //             " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as
+    // TIMESTAMP) ;",
+    //             dateToSqlFormat(startDate), dateToSqlFormat(endDate)));
+
+    addQueryTask(
+        out,
+        "task_instance.csv",
+        dateRange == null
+            ? "select * from task_instance;"
+            : "select * from task_instance "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
+
     // Airflow v1.6.0
-    addFullTable(out, "dag_run.csv", "select * from dag_run;");
+    addQueryTask(
+        out,
+        "dag_run.csv",
+        dateRange == null
+            ? "select * from dag_run;"
+            : "select * from dag_run "
+                + String.format(
+                    " where end_date >= CAST( '%s' as TIMESTAMP) and end_date < CAST( '%s' as TIMESTAMP) ;",
+                    dateToSqlFormat(dateRange.getLeft()), dateToSqlFormat(dateRange.getRight())));
 
     // Airflow v1.10.7
     // analog of DAG's python definition in json
-    addFullTable(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "serialized_dag.csv", "select * from serialized_dag;", TaskCategory.OPTIONAL);
     // Airflow v1.10.10
-    addFullTable(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+    addQueryTask(out, "dag_code.csv", "select * from dag_code;", TaskCategory.OPTIONAL);
+  }
+
+  @Nullable
+  private ZonedIntervalIterable fromAirflowConnectorArguments(ConnectorArguments arguments) {
+    Pair<ZonedDateTime, ZonedDateTime> dateRange = dateRange(arguments);
+    if (dateRange == null) {
+      LOG.info("Date ranges was not specified. Generate full table queries.");
+      return null;
+    }
+
+    LOG.info(
+        "Date range for query generation from {} to {} exclusive and increments of one day.",
+        dateRange.getLeft(),
+        dateRange.getRight());
+    return ZonedIntervalIterableGenerator.forDateRangeWithIntervalDuration(
+        dateRange.getLeft(), dateRange.getRight(), Duration.ofDays(1));
+  }
+
+  @Nullable
+  private Pair<ZonedDateTime, ZonedDateTime> dateRange(ConnectorArguments arguments) {
+    boolean isDateRangeSpecified =
+        arguments.getStartDate() != null || arguments.getLookbackDays() != null;
+    if (!isDateRangeSpecified) {
+      return null;
+    }
+
+    ZonedDateTime startDate;
+    ZonedDateTime endDate;
+    if (arguments.getLookbackDays() != null) {
+      endDate = arguments.getStartDate(ZonedDateTime.now(ZoneOffset.UTC));
path: dumper/app/src/main/java/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
user: login: vladislav-sidorovich
body: I remove `lookback` param for now
created_at: 2025-03-25T09:32:47Z
author_association: MEMBER

PR comments:
user: login: kpiotrowski
created_at: 2025-03-07T09:04:11Z
body: -start-date=x --lookback-days=y -> date range: [x-y, x)

This usage is not intuitive, at least for me. Usually when I provide start date I want all the data to be later than this. I think `end-date` works better with a `lookback-days` because lookback refers to searching N days in the past.
author_association: CONTRIBUTOR
user: login: vladislav-sidorovich
created_at: 2025-03-07T10:37:51Z
body: > -start-date=x --lookback-days=y -> date range: [x-y, x)
> 
> This usage is not intuitive, at least for me. Usually when I provide start date I want all the data to be later than this. I think `end-date` works better with a `lookback-days` because lookback refers to searching N days in the past.

I do the calculation and log the actual time range to show the date range explicitly. I agree that `start-date` might be confusing. However the logic behind it is the following:  I chose `X` date in the calendar and go back for `Y` days, as the result it will be a date range.  In other words, there is an ability to move `now`. 
With such logic go back from `end` is confusing as well. 

I'm open to ideas.

There is a good example: https://grafana.com/docs/grafana/latest/dashboards/use-dashboards/#set-dashboard-time-range  
For example we can rename `start` -> `from` and `end` - `to`. So, it will be: `from` date `X` look back for `Y` days.

does it look better? 
author_association: MEMBER
user: login: vladislav-sidorovich
created_at: 2025-03-25T09:06:57Z
body: I will keep only `start` and `end` date parameters for now. `lookback-days` or similar parameter will be introduced in a separate PR.
Eventually these params should be applied to all connectors (metadata and query logs). 
author_association: MEMBER