Repository: apache/pinot

Issue #8219: consumption halted on realtime table when accessing an offset that has been already deleted from Kafka
Created: 2022-02-17T19:30:30Z
the realtime consumption of records on realtime tables stops when given realtime table need to access an offset that has been already wiped by kafka, this can happen because the retention policy in the kafka side is lesser than how often we commit segments in pinot, however when this happens, there's no easy way for us to recover and start consumption again on the realtime table, the log message in the pinot-servers shows the following:
Fetch position xxxx FetchPosition xxx is out of range for partition resetting offset xxxx
consumer plugin: org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory
logs:
Consumed 0 events from (rate:0.0/s), currentOffset=5011308264, numRowsConsumedSoFar=0, numRowsIndexedSoFar=0
[Consumer clientId=consumer-null-12, groupId=null] Seeking to offset 5011308265 for partition xx-xx-xx-xx-3
Consumed 0 events from (rate:0.0/s), currentOffset=5110008164, numRowsConsumedSoFar=0, numRowsIndexedSoFar=0
[Consumer clientId=consumer-null-1, groupId=null] Seeking to offset 5110008165 for partition xx-xx-xx-xx-1
[Consumer clientId=consumer-null-12, groupId=null] Fetch position FetchPosition{offset=5011308265, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[xxx (id: 0 rack: us], epoch=6}} is out of range for partition xx-xx-xx-xx-3, resetting offset
[Consumer clientId=consumer-null-31, groupId=null] Fetch position FetchPosition{offset=4849504882, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[xxx (id: 2 rack: us)], epoch=5}} is out of range for partition xx-xx-xx-xx-5, resetting offset
[Consumer clientId=consumer-null-12, groupId=null] Resetting offset for partition xx-xx-xx-xx-3 to position FetchPosition{offset=546724258, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[xxx (id: 0 rack: us)], epoch=6}}.
[Consumer clientId=consumer-null-31, groupId=null] Resetting offset for partition xx-xx-xx-xx-5 to position FetchPosition{offset=530075133, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[xxx (id: 2 rack: us)], epoch=5}}.
Consumed 0 events from (rate:0.0/s), currentOffset=5063423275, numRowsConsumedSoFar=0, numRowsIndexedSoFar=0

There should be a way to recover from this scenario without having to recreate the table or having to manually update the zk options for the consuming segment which is hard to find in the zk side.

Duplicate issue #8717: Data not getting consumed from all Kafka partitions
Created: 2022-05-17T07:01:34Z
Issue currently being faced one of our users.
Topic has 3 partitions

Data getting pushed to all 3 partitions

When table was created, it was consuming data from all 3 partitions

However, after 24 hours, it has been consuming data only from partition 1. partition 0 an 2 segments show in the CONSUMING state and their metadata is also present in PROPERTY_STORE.

Resetting the stuck segments didn't work as well.

Here is what segments status looks like
{
  "id": "tSCalibrationAttempt_REALTIME",
  "simpleFields": {
    "BATCH_MESSAGE_MODE": "false",
    "IDEAL_STATE_MODE": "CUSTOMIZED",
    "INSTANCE_GROUP_TAG": "tSCalibrationAttempt_REALTIME",
    "MAX_PARTITIONS_PER_INSTANCE": "1",
    "NUM_PARTITIONS": "13",
    "REBALANCE_MODE": "CUSTOMIZED",
    "REPLICAS": "1",
    "STATE_MODEL_DEF_REF": "SegmentOnlineOfflineStateModel",
    "STATE_MODEL_FACTORY_NAME": "DEFAULT"
  },
  "mapFields": {
    "tSCalibrationAttempt__0__0__20220505T1656Z": {
      "Server_pinot-server-0.pinot-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "tSCalibrationAttempt__0__1__20220506T1656Z": {
      "Server_pinot-server-0.pinot-server-headless.pinot.svc.cluster.local_8098": "CONSUMING"
    },
    "tSCalibrationAttempt__1__0__20220505T1656Z": {
      "Server_pinot-server-1.pinot-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "tSCalibrationAttempt__1__1__20220506T1656Z": {
      "Server_pinot-server-1.pinot-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "tSCalibrationAttempt__1__2__20220509T0456Z": {
      "Server_pinot-server-1.pinot-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "tSCalibrationAttempt__1__3__20220510T0456Z": {
      "Server_pinot-server-1.pinot-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "tSCalibrationAttempt__1__4__20220511T0456Z": {
      "Server_pinot-server-1.pinot-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "tSCalibrationAttempt__1__5__20220512T0456Z": {
      "Server_pinot-server-1.pinot-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "tSCalibrationAttempt__1__6__20220513T0456Z": {
      "Server_pinot-server-1.pinot-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "tSCalibrationAttempt__1__7__20220514T0456Z": {
      "Server_pinot-server-1.pinot-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "tSCalibrationAttempt__1__8__20220516T0657Z": {
      "Server_pinot-server-1.pinot-server-headless.pinot.svc.cluster.local_8098": "CONSUMING"
    },
    "tSCalibrationAttempt__2__0__20220505T1656Z": {
      "Server_pinot-server-0.pinot-server-headless.pinot.svc.cluster.local_8098": "ONLINE"
    },
    "tSCalibrationAttempt__2__1__20220506T1656Z": {
      "Server_pinot-server-0.pinot-server-headless.pinot.svc.cluster.local_8098": "CONSUMING"
    }
  },
  "listFields": {}
}

Data is not getting consumed from partition 0 and 2. For Partition 2, here's what metadata looks like
{
  "id": "tSCalibrationAttempt__2__0__20220505T1656Z",
  "simpleFields": {
    "segment.crc": "2706892624",
    "segment.creation.time": "1651769803410",
    "segment.download.url": "http://pinot-controller-0.pinot-controller-headless.pinot.svc.cluster.local:9000/segments/tSCalibrationAttempt/tSCalibrationAttempt__2__0__20220505T1656Z",
    "segment.end.time": "1651847119229",
    "segment.flush.threshold.size": "100000",
    "segment.index.version": "v3",
    "segment.realtime.endOffset": "819",
    "segment.realtime.numReplicas": "1",
    "segment.realtime.startOffset": "746",
    "segment.realtime.status": "DONE",
    "segment.start.time": "1651648021433",
    "segment.time.unit": "MILLISECONDS",
    "segment.total.docs": "73"
  },
  "mapFields": {},
  "listFields": {}
}

{
  "id": "tSCalibrationAttempt__2__1__20220506T1656Z",
  "simpleFields": {
    "segment.creation.time": "1651856206431",
    "segment.flush.threshold.size": "10000",
    "segment.realtime.numReplicas": "1",
    "segment.realtime.startOffset": "819",
    "segment.realtime.status": "IN_PROGRESS"
  },
  "mapFields": {},
  "listFields": {}
}


Table config:
{
  "REALTIME": {
    "tableName": "tSCalibrationAttempt_REALTIME",
    "tableType": "REALTIME",
    "segmentsConfig": {
      "schemaName": "tSCalibrationAttempt",
      "replication": "1",
      "timeColumnName": "coeff.calTimestamp",
      "allowNullTimeValue": false,
      "replicasPerPartition": "1",
      "completionConfig": {
        "completionMode": "DOWNLOAD"
      }
    },
    "tenants": {
      "broker": "DefaultTenant",
      "server": "DefaultTenant",
      "tagOverrideConfig": {}
    },
    "tableIndexConfig": {
      "invertedIndexColumns": [],
      "streamConfigs": {
        "streamType": "kafka",
        "stream.kafka.topic.name": "tscalibrationattempt",
        "stream.kafka.broker.list": "",
        "stream.kafka.consumer.type": "lowlevel",
        "stream.kafka.consumer.prop.auto.offset.reset": "largest",
        "stream.kafka.schema.registry.url": "",
        "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka20.KafkaConsumerFactory",
        "stream.kafka.decoder.prop.schema.registry.rest.url": "",
        "sasl.mechanism": "PLAIN",
        "stream.kafka.decoder.prop.basic.auth.credentials.source": "USER_INFO",
        "stream.kafka.decoder.prop.schema.registry.basic.auth.user.info": "",
        "security.protocol": "SASL_SSL",
        "sasl.jaas.config": "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"\" password=\"\";",
        "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder",
        "realtime.segment.flush.threshold.rows": "0",
        "realtime.segment.flush.threshold.time": "24h",
        "realtime.segment.flush.threshold.segment.size": "150M"
      },
      "rangeIndexColumns": [
        "coeff.calTimestamp"
      ],
      "rangeIndexVersion": 2,
      "autoGeneratedInvertedIndex": false,
      "createInvertedIndexDuringSegmentGeneration": false,
      "sortedColumn": [
        "deviceId"
      ],
      "bloomFilterColumns": [],
      "loadMode": "MMAP",
      "onHeapDictionaryColumns": [],
      "varLengthDictionaryColumns": [],
      "enableDefaultStarTree": false,
      "enableDynamicStarTreeCreation": false,
      "aggregateMetrics": false,
      "nullHandlingEnabled": false
    },
    "metadata": {},
    "quota": {},
    "routing": {
      "instanceSelectorType": "strictReplicaGroup"
    },
    "query": {},
    "ingestionConfig": {
      "complexTypeConfig": {
        "fieldsToUnnest": [
          "coeff"
        ]
      }
    },
    "isDimTable": false
  }
}
Update:
Once the problematic segments (tSCalibrationAttempt__2__1__20220506T1656Z and tSCalibrationAttempt__0__1__20220506T1656Z) are deleted via API call (DELETE /segments/{tableName}/{segmentName}) , the consumption resumes succesfully for partition 2 but no segments still created for partition 0.

Closing comment (2022-05-17T19:46:47Z): @KKcorps this sounds similar to #8219. This had happened because the server was down/consumption was lagging and the offset expired before it could be consumed. More details in the PRs linked to the issue.
Can you check with them for logs, and if you can see the same log lines as mentioned in the issue?