Project: springxd
Issue XD-2050: HDFS sink Partition Path causing writes to be slower in certain cases
I have not tested this on M7 but I believe it is the case with latest release as well. Stream definition 1: stream create logIngestion --definition "rabbit --queues=demo --host= | script --location=linemerge.groovy | hdfs --rollover=10M --idleTimeout=10000 --fileUuid=true --directory=/data/loganalysis --partitionPath=path(payload.split('\u0001')[1],dateFormat('yyyy/MM/dd/HH',payload.split('\u0001')[0],'yyyyMMddHHmmss'))" we noticed this was causing writes to be slower Stream definition 2: stream create logIngestion --definition "rabbit --queues=demo --host= | script --location=linemerge.groovy | hdfs --rollover=10M --idleTimeout=10000 --fileUuid=true --directory=/data/loganalysis " but this definition caused the writes to be much faster. Please note this was just a one time test I did and not reproduced multiple times. Janne also seems to have reproduced this in another use case. Thanks Girish
Story points: 8