auphong2707/dl4se-code-review-dataset huggingface dataset:
Source:
public IndexIngestionSpec( @JsonProperty("dataSchema") DataSchema dataSchema, @JsonProperty("ioConfig") IndexIOConfig ioConfig, @JsonProperty("tuningConfig") IndexTuningConfig tuningConfig ) { super(dataSchema, ioConfig, tuningConfig); this.dataSchema = dataSchema; this.ioConfig = ioConfig; this.tuningConfig = tuningConfig == null ? new IndexTuningConfig() : tuningConfig; START if (this.ioConfig.isAppendToExisting() && this.tuningConfig.isForceGuaranteedRollup()) { END log.warn("Perfect rollup is not guaranteed when appending to existing data sources." + " forceGuaranteedRollup flag is ignored."); } } Avoid warnings on construction, since constructors get called on deserialization in many contexts. Also, is this useful, given there is another check is the task itself?

Target:
public IndexIngestionSpec( @JsonProperty("dataSchema") DataSchema dataSchema, @JsonProperty("ioConfig") IndexIOConfig ioConfig, @JsonProperty("tuningConfig") IndexTuningConfig tuningConfig ) { super(dataSchema, ioConfig, tuningConfig); this.dataSchema = dataSchema; this.ioConfig = ioConfig; this.tuningConfig = tuningConfig == null ? new IndexTuningConfig() : tuningConfig; }
Source:
public void reset() { START hasParsedHeader = !firstRowIsHeader; END } This could just be `hasParsedHeader = false` right? Seems simpler

Target:
public void reset() { hasParsedHeader = false; }
Source:
public TaskAnnouncementTest() { TestUtils testUtils = new TestUtils(); jsonMapper = testUtils.getTestObjectMapper(); final File baseDir = new File("lol"); baseDir.mkdir(); START baseDir.deleteOnExit(); END } What's this for? Could it be a `@Rule TemporaryFolder` instead? That's preferred for any temporary directories or files needed by tests.

Target:
public TaskAnnouncementTest() { TestUtils testUtils = new TestUtils(); jsonMapper = testUtils.getTestObjectMapper(); }
Source:
public LocalFirehoseFactory( @JsonProperty("baseDir") File baseDir, @JsonProperty("filter") String filter, // Backwards compatible @JsonProperty("parser") StringInputRowParser parser ) { super( START FileUtils.listFiles( END Preconditions.checkNotNull(baseDir).getAbsoluteFile(), new WildcardFileFilter(filter), TrueFileFilter.INSTANCE ) ); this.baseDir = baseDir; this.filter = filter; this.parser = parser; log.info("Initialized with " + getObjects() + " files"); } This shouldn't happen in construction. Firehose factories get constructed semi-frequently (anytime Task objects are deserialized) and that doesn't necessarily mean they will really be used. Anything that hits the disk or network should be deferred until the firehose is connected.

Target:
public LocalFirehoseFactory( @JsonProperty("baseDir") File baseDir, @JsonProperty("filter") String filter, // Backwards compatible @JsonProperty("parser") StringInputRowParser parser ) { this.baseDir = baseDir; this.filter = filter; this.parser = parser; }
Source:
public TaskSerdeTest() { TestUtils testUtils = new TestUtils(); jsonMapper = testUtils.getTestObjectMapper(); for (final Module jacksonModule : new FirehoseModule().getJacksonModules()) { jsonMapper.registerModule(jacksonModule); } final File baseDir = new File("lol"); baseDir.mkdir(); START baseDir.deleteOnExit(); END } Could this be a `@Rule TemporaryFolder` instead?

Target:
public TaskSerdeTest() { TestUtils testUtils = new TestUtils(); jsonMapper = testUtils.getTestObjectMapper(); for (final Module jacksonModule : new FirehoseModule().getJacksonModules()) { jsonMapper.registerModule(jacksonModule); } }
Source:
public Function<Result<TimeseriesResultValue>, Result<TimeseriesResultValue>> makePostComputeManipulatorFn( TimeseriesQuery query, MetricManipulationFn fn ) { boolean shouldFinalize = MetricManipulatorFns.finalizing().equals(fn); START return makeComputeManipulatorFn(query, fn, shouldFinalize); END } The equals check is brittle, I think it'd be better to do a check against a query context key. Or alternatively, do something like groupBy does, and use `mergeResults` to modify the query to remove the post-aggregators. Then, everything downstream of mergeResults on the outermost node (typically the broker) will ignore the post-aggregators. That could be done by overriding `doRun` in the ResultMergeQueryRunner.

Target:
public Function<Result<TimeseriesResultValue>, Result<TimeseriesResultValue>> makePostComputeManipulatorFn( TimeseriesQuery query, MetricManipulationFn fn ) { return makeComputeManipulatorFn(query, fn, true); }
Source:
public List<String> getNames() { final Iterable<DataSourceWithSegmentSpec> sourceSpecs = query.getDataSources(); return StreamSupport.stream(sourceSpecs.spliterator(), false) .flatMap(spec -> spec.getDataSource().getNames().stream()) START .collect(Collectors.toList()); END } What if there are duplicates in this stream?

Target:
public List<String> getNames() { return query.getDataSources().stream() .flatMap(spec -> spec.getDataSource().getNames().stream()) .collect(Collectors.toList()); }
Source:
public ValueMatcher makeValueMatcher(final LongColumnSelector selector, final String value) { Long matchVal = DimensionHandlerUtils.convertObjectToLong(value); if (matchVal == null) { START matchVal = DimensionHandlerUtils.getIntegralFromFloatString(value); END } if (matchVal == null) { return BooleanValueMatcher.of(false); } final long matchValLong = matchVal; return new ValueMatcher() { @Override public boolean matches() { return selector.get() == matchValLong; } }; } Could this method call and check be done inside `DimensionHandlerUtils.convertObjectToLong()`?

Target:
public ValueMatcher makeValueMatcher(final LongColumnSelector selector, final String value) { final Long matchVal = DimensionHandlerUtils.convertObjectToLong(value); if (matchVal == null) { return BooleanValueMatcher.of(false); } final long matchValLong = matchVal; return new ValueMatcher() { @Override public boolean matches() { return selector.get() == matchValLong; } }; }
Source:
private BigDecimal getBigDecimalLowerBoundFromFloatString(String floatStr) { BigDecimal convertedBD; try { convertedBD = new BigDecimal(floatStr); } catch (NumberFormatException nfe) { return null; } if (lowerStrict) { START convertedBD = convertedBD.setScale(0, RoundingMode.FLOOR); END } else { convertedBD = convertedBD.setScale(0, RoundingMode.CEILING); } return convertedBD; } Need not complicate control flow, suggested `return convertedBD.setScale(0, RoundingMode.FLOOR);` (and below)

Target:
private BigDecimal getBigDecimalLowerBoundFromFloatString(String floatStr) { BigDecimal convertedBD; try { convertedBD = new BigDecimal(floatStr); } catch (NumberFormatException nfe) { return null; } if (lowerStrict) { return convertedBD.setScale(0, RoundingMode.FLOOR); } else { return convertedBD.setScale(0, RoundingMode.CEILING); } }
Source:
public ValueMatcher makeValueMatcher(final LongColumnSelector selector, final String value) { START Long matchVal = DimensionHandlerUtils.convertObjectToLong(value); END if (matchVal == null) { return BooleanValueMatcher.of(false); } final long matchValLong = matchVal; return new ValueMatcher() { @Override public boolean matches() { return selector.get() == matchValLong; } }; } Could have stayed final.

Target:
public ValueMatcher makeValueMatcher(final LongColumnSelector selector, final String value) { final Long matchVal = DimensionHandlerUtils.convertObjectToLong(value); if (matchVal == null) { return BooleanValueMatcher.of(false); } final long matchValLong = matchVal; return new ValueMatcher() { @Override public boolean matches() { return selector.get() == matchValLong; } }; }
Source:
public Object convertUnsortedEncodedKeyComponentToActualArrayOrList(Long key, boolean asList) { START return Lists.newArrayList(key); END } ImmutableList.of may be cheaper. It just stores the `key` as a field.

Target:
public Object convertUnsortedEncodedKeyComponentToActualArrayOrList(Long key, boolean asList) { return ImmutableList.of(key); }
Source:
protected int tryAcquireShared(int ignore) { START return getState() >= STARTED ? 1 : -1; END } Should this be `getState() > STARTED`? It looks like with `>=`, awaitStarted would be able to proceed to the state check when the state is not yet some kind of `START_EXITED_*`, and so it might fail when it should have succeeded.

Target:
protected int tryAcquireShared(int ignore) { return getState() > STARTED ? 1 : -1; }
Source:
public String toString() { return "TimeseriesQuery{" + "dataSource='" + getDataSource() + '\'' + ", querySegmentSpec=" + getQuerySegmentSpec() + START ", virtualColumns=" + virtualColumns + END ", descending=" + isDescending() + ", dimFilter=" + dimFilter + ", granularity='" + granularity + '\'' + ", aggregatorSpecs=" + aggregatorSpecs + ", postAggregatorSpecs=" + postAggregatorSpecs + ", context=" + getContext() + '}'; } virtualColumns and descending go in different order elsewhere in this class

Target:
public String toString() { return "TimeseriesQuery{" + "dataSource='" + getDataSource() + '\'' + ", querySegmentSpec=" + getQuerySegmentSpec() + ", descending=" + isDescending() + ", virtualColumns=" + virtualColumns + ", dimFilter=" + dimFilter + ", granularity='" + granularity + '\'' + ", aggregatorSpecs=" + aggregatorSpecs + ", postAggregatorSpecs=" + postAggregatorSpecs + ", context=" + getContext() + '}'; }
Source:
public ImmutableBitmap union(Iterable<ImmutableBitmap> b) throws ClassCastException { START if (b instanceof List) { END final List<ImmutableBitmap> bitmapList = (List<ImmutableBitmap>) b; if (bitmapList.isEmpty()) { return makeEmptyImmutableBitmap(); } else if (bitmapList.size() == 1) { return Iterables.getOnlyElement(b); } } return new WrappedImmutableConciseBitmap(ImmutableConciseSet.union(unwrap(b))); } Can check for `Collection` here

Target:
public ImmutableBitmap union(Iterable<ImmutableBitmap> b) throws ClassCastException { if (b instanceof Collection) { final Collection<ImmutableBitmap> bitmapList = (Collection<ImmutableBitmap>) b; final int size = bitmapList.size(); if (size == 0) { return makeEmptyImmutableBitmap(); } else if (size == 1) { return Iterables.getOnlyElement(b); } } return new WrappedImmutableConciseBitmap(ImmutableConciseSet.union(unwrap(b))); }
Source:
byte[] getPrototypeClassBytecode() throws IOException { if (prototypeClassBytecode == null) { ClassLoader cl = prototypeClass.getClassLoader(); START InputStream prototypeClassBytecodeStream = cl.getResourceAsStream(prototypeClassBytecodeName + ".class"); END prototypeClassBytecode = ByteStreams.toByteArray(prototypeClassBytecodeStream); } return prototypeClassBytecode; } Need to close the stream.

Target:
byte[] getPrototypeClassBytecode() throws IOException { if (prototypeClassBytecode == null) { ClassLoader cl = prototypeClass.getClassLoader(); try (InputStream prototypeClassBytecodeStream = cl.getResourceAsStream(prototypeClassBytecodeName + ".class")) { prototypeClassBytecode = ByteStreams.toByteArray(prototypeClassBytecodeStream); } } return prototypeClassBytecode; }
Source:
START } catch (Exception e) { END throw new RuntimeException("Cannot access Unsafe methods", e); } The catch block needs to be moved to one line below.

Target:
catch (Exception e) { throw new RuntimeException("Cannot access Unsafe methods", e); }
Source:
START public boolean determineApplyLimitPushDown() END { final boolean forceLimitPushDown = validateAndGetForceLimitPushDown(); if (limitSpec instanceof DefaultLimitSpec) { DefaultLimitSpec defaultLimitSpec = (DefaultLimitSpec) limitSpec; // If only applying an orderby without a limit, don't try to push down if (defaultLimitSpec.getLimit() == Integer.MAX_VALUE) { return false; } if (forceLimitPushDown) { return true; } // If the sorting order only uses columns in the grouping key, we can always push the limit down // to the buffer grouper without affecting result accuracy boolean sortHasNonGroupingFields = DefaultLimitSpec.sortingOrderHasNonGroupingFields( limitSpec, getDimensions() ); return !sortHasNonGroupingFields; } return false; } i think this can be implemented in one line `return validateAndGetForceLimitPushDown() || DefaultLimitSpec.sortingOrderHasNonGroupingFields(..);` you might have to add the `defaultLimitSpec.getLimit() == Integer.MAX_VALUE` check inside `DefaultLimitSpec.sortingOrderHasNonGroupingFields(..)`

Target:
public boolean determineApplyLimitPushDown() { final boolean forceLimitPushDown = validateAndGetForceLimitPushDown(); if (limitSpec instanceof DefaultLimitSpec) { DefaultLimitSpec defaultLimitSpec = (DefaultLimitSpec) limitSpec; // If only applying an orderby without a limit, don't try to push down if (defaultLimitSpec.getLimit() == Integer.MAX_VALUE) { return false; } if (forceLimitPushDown) { return true; } // If the sorting order only uses columns in the grouping key, we can always push the limit down // to the buffer grouper without affecting result accuracy boolean sortHasNonGroupingFields = DefaultLimitSpec.sortingOrderHasNonGroupingFields( (DefaultLimitSpec) limitSpec, getDimensions() ); return !sortHasNonGroupingFields; } return false; }
Source:
public void reset() { size = 0; growthCount = 0; // clear the used bits of both buffers for (int i = 0; i < maxBuckets; i++) { subHashTableBuffers[0].put(i * bucketSizeWithHash, (byte) 0); START subHashTableBuffers[1].put(i * bucketSizeWithHash, (byte) 0); END } tableBuffer = subHashTableBuffers[0]; } this one is not needed and will automatically be zeroed out in adjustTableWhenFull(..) when needed.

Target:
public void reset() { size = 0; growthCount = 0; // clear the used bits of the first buffer for (int i = 0; i < maxBuckets; i++) { subHashTableBuffers[0].put(i * bucketSizeWithHash, (byte) 0); } tableBuffer = subHashTableBuffers[0]; }
Source:
private void sendToKafka(final String topic, MemoryBoundLinkedBlockingQueue<String> recordQueue) { ObjectContainer<String> objectToSend; try { START while((objectToSend = recordQueue.take()) != null) { END producer.send(new ProducerRecord<String, String>(topic, objectToSend.getData()), producerCallback); } } catch (InterruptedException e) { log.warn(e, "Failed to take record from queue!"); } } `take` would never be null, right? Did you mean poll?

Target:
private void sendToKafka(final String topic, MemoryBoundLinkedBlockingQueue<String> recordQueue) { ObjectContainer<String> objectToSend; try { while (true) { objectToSend = recordQueue.take(); producer.send(new ProducerRecord<>(topic, objectToSend.getData()), producerCallback); } } catch (InterruptedException e) { log.warn(e, "Failed to take record from queue!"); } }
Source:
public int hashCode() { START return 1; END } Suggested `getClass().hashCode()` as something "more random"

Target:
public int hashCode() { return getClass().hashCode(); }
Source:
public static ValueType fromString(String name) { if (name == null) { START return ValueType.STRING; END } return valueOf(name.toUpperCase()); } Better to return `null` here and let callers deal with `null` however they want. This will default to STRING in _all_ cases of deserializing ValueType, even unforeseen ones where that might not be ideal behavior.

Target:
public static ValueType fromString(String name) { if (name == null) { return null; } return valueOf(name.toUpperCase()); }
Source:
Outer(int fileNum, File outFile, int maxLength) throws FileNotFoundException { this.fileNum = fileNum; this.outFile = outFile; this.maxLength = maxLength; FileOutputStream outStream = new FileOutputStream(outFile); this.channel = outStream.getChannel(); START closer.register(outStream); END closer.register(channel); } can you move the closer to the same place the file output stream is created? ```java final FileOutputStream outStream = closer.register(new FileOutputStream(outFile)); ``` should work

Target:
Outer(int fileNum, File outFile, int maxLength) throws FileNotFoundException { this.fileNum = fileNum; this.outFile = outFile; this.maxLength = maxLength; FileOutputStream outStream = closer.register(new FileOutputStream(outFile)); this.channel = closer.register(outStream.getChannel()); }
Source:
public int get(int index) { START return 0; END } Have you intentionally omitted index check?

Target:
public int get(int index) { // Skip range check in production, assume "index" was 0 like it really should have been. assert index == 0; return 0; }
Source:
public static int findOpenPort(int basePort) { START final int startPort = basePort == -1 ? -1 : rnd.nextInt(0x7fff) + basePort; END return findOpenPortFrom(startPort); } let's make this check for all negatives instead of just -1

Target:
public static int findOpenPort(int basePort) { final int startPort = basePort < 0 ? -1 : rnd.nextInt(0x7fff) + basePort; return findOpenPortFrom(startPort); }
Source:
public void close() throws IOException { START flush(); END started.set(false); exec.shutdown(); } need to make sure that only one call to flush is in the queue maybe synchronize around start ?

Target:
public void close() throws IOException { synchronized (started) { flush(); exec.shutdown(); started.set(false); } }
Source:
public Response getVersion() { START CacheScheduler.CacheState cacheState = factory.entry.getCacheState(); END if (cacheState instanceof CacheScheduler.NoCache) { return Response.status(Response.Status.NOT_FOUND).build(); } else { String version = ((CacheScheduler.VersionedCache) cacheState).getVersion(); return Response.ok(ImmutableMap.of("version", version)).build(); } } suggest `final`

Target:
public Response getVersion() { final CacheScheduler.CacheState cacheState = factory.entry.getCacheState(); if (cacheState instanceof CacheScheduler.NoCache) { return Response.status(Response.Status.NOT_FOUND).build(); } else { String version = ((CacheScheduler.VersionedCache) cacheState).getVersion(); return Response.ok(ImmutableMap.of("version", version)).build(); } }
Source:
START private void createDanglingObjectHandler() END { poolOfString.take(); } can we remove this method and simply have the poolsOfString.take() where this method is called?

Target:
private String createDanglingObjectHandler() { return poolOfString.take().get(); }
Source:
private static String makeId(String id, IndexIngestionSpec ingestionSchema) { START return id != null ? id : String.format("index_%s_%s", makeDataSource(ingestionSchema), new DateTime().toString()); END } Don't need `toString()`

Target:
private static String makeId(String id, IndexIngestionSpec ingestionSchema) { return id != null ? id : String.format("index_%s_%s", makeDataSource(ingestionSchema), new DateTime()); }
Source:
public void close() throws IOException { START CloseQuietly.close(jsonParser); END it = Iterators.emptyIterator(); } If `close()` throws IOException, why consume possible IOException in `jsonParser`?

Target:
public void close() throws IOException { if (jsonParser != null) { jsonParser.close(); } it = Iterators.emptyIterator(); }
Source:
public Callable<String> getCachePopulator( final String id, final StaticMapExtractionNamespace extractionNamespace, final String lastVersion, final Map<String, String> swap ) { return new Callable<String>() { @Override public String call() throws Exception { if (lastVersion != null) { // Throwing AssertionError, because NamespaceExtractionCacheManager doesn't suppress Errors and will stop // trying to update the cache periodically. START throw new AssertionError( END "StaticMapExtractionNamespaceCacheFactory could only be configured for a namespace which is scheduled " + "to be updated once, not periodically. Last version: `" + lastVersion + "`"); } else { swap.putAll(extractionNamespace.getMap()); return version; } } }; } Can this throw in `getCachePopulator` instead of in the callable? that way there's a failure immediately.

Target:
public Callable<String> getCachePopulator( final String id, final StaticMapExtractionNamespace extractionNamespace, final String lastVersion, final Map<String, String> swap ) { if (lastVersion != null) { // Throwing AssertionError, because NamespaceExtractionCacheManager doesn't suppress Errors and will stop trying // to update the cache periodically. throw new AssertionError( "StaticMapExtractionNamespaceCacheFactory could only be configured for a namespace which is scheduled " + "to be updated once, not periodically. Last version: `" + lastVersion + "`"); } return new Callable<String>() { @Override public String call() throws Exception { swap.putAll(extractionNamespace.getMap()); return version; } }; }
Source:
public Aggregator factorize(ColumnSelectorFactory metricFactory) { return new DoubleFirstAggregator( name, metricFactory.makeFloatColumnSelector(fieldName), START metricFactory.makeLongColumnSelector(Column.TIME_COLUMN_NAME) END ); } minor nit: can we pass time column selector before field selector to be consistent with buffered aggregator ?

Target:
public Aggregator factorize(ColumnSelectorFactory metricFactory) { return new DoubleFirstAggregator( name, metricFactory.makeLongColumnSelector(Column.TIME_COLUMN_NAME), metricFactory.makeFloatColumnSelector(fieldName) ); }
Source:
public PrefetchedLoadingLookupFactory( @JsonProperty("dataFetcher") PrefetchableFetcher prefetchableFetcher, @JsonProperty("loadingCacheSpec") LoadingCache<String, String> loadingCache, @JsonProperty("reverseLoadingCacheSpec") LoadingCache<String, List<String>> reverseLoadingCache ) { START super(prefetchableFetcher, loadingCache, reverseLoadingCache, new PrefetchedLoadingLookup(prefetchableFetcher,loadingCache,reverseLoadingCache)); END } this instead of super?

Target:
public PrefetchedLoadingLookupFactory( @JsonProperty("dataFetcher") PrefetchableFetcher prefetchableFetcher, @JsonProperty("loadingCacheSpec") LoadingCache<String, String> loadingCache, @JsonProperty("reverseLoadingCacheSpec") LoadingCache<String, List<String>> reverseLoadingCache ) { this(prefetchableFetcher, loadingCache, reverseLoadingCache, new PrefetchedLoadingLookup(prefetchableFetcher,loadingCache,reverseLoadingCache)); }
Source:
public Map<String, String> parse(String input) { final Map<String, Object> inner = delegate.parse(input); final String k = Preconditions.checkNotNull( inner.get(key), "Key column [%s] missing data in line [%s]", key, input ).toString(); // Just in case is long final Object val = inner.get(value); if (val == null) { // Skip nulls START return null; END } return ImmutableMap.of(k, val.toString()); } Since `Parser.parse()` is not marked as Nullable, maybe saner to return `emptyMap()` here and remove the check in `MapPopulator`?

Target:
public Map<String, String> parse(String input) { final Map<String, Object> inner = delegate.parse(input); final String k = Preconditions.checkNotNull( inner.get(key), "Key column [%s] missing data in line [%s]", key, input ).toString(); // Just in case is long final Object val = inner.get(value); if (val == null) { // Skip null or missing values, treat them as if there were no row at all. return ImmutableMap.of(); } return ImmutableMap.of(k, val.toString()); }
Source:
public void testThrowParseExceptions(){ InputRow in = new MapBasedInputRow( timestamp, dims, event ); AggregatorFactory[] aggregatorFactories = new AggregatorFactory[] { new DoubleSumAggregatorFactory("agg_non_existing", "agg_non_existing_in"), new DoubleSumAggregatorFactory("m1out", "m1"), new LongSumAggregatorFactory("m2out", "m2"), new HyperUniquesAggregatorFactory("m3out", "m3"), new LongSumAggregatorFactory("unparseable", "m3") // Unparseable from String to Long }; try { InputRowSerde.toBytes(in, aggregatorFactories, true); Assert.fail("Should have thrown parseException"); START } catch (ParseException e){ END // expected } } Better to use the `org.junit.rules.ExpectedException` `@Rule` or use `@Test(expected = ParseException.class)`

Target:
public void testThrowParseExceptions() { InputRow in = new MapBasedInputRow( timestamp, dims, event ); AggregatorFactory[] aggregatorFactories = new AggregatorFactory[]{ new DoubleSumAggregatorFactory("agg_non_existing", "agg_non_existing_in"), new DoubleSumAggregatorFactory("m1out", "m1"), new LongSumAggregatorFactory("m2out", "m2"), new HyperUniquesAggregatorFactory("m3out", "m3"), new LongSumAggregatorFactory("unparseable", "m3") // Unparseable from String to Long }; InputRowSerde.toBytes(in, aggregatorFactories, true); }
Source:
public void read1() { START for (int i = 0; i < 500000; i++) { END sum += d1.get(i); } } should this count be parameterized?

Target:
public void read1() { for (int i = 0; i < values; i++) { sum += d1.get(i); } }
Source:
public static Expr binaryOp(BinaryOpExprBase binary, Expr left, Expr right) { try { return binary.getClass() .getDeclaredConstructor(String.class, Expr.class, Expr.class) .newInstance(binary.op, left, right); } catch (Exception e) { START log.info(e, "failed to rewrite expression " + binary); END return binary; // best effort.. keep it working } } Do we expect this to happen "normally" or is this really a problem indicating a bug? If it's going to happen "normally" it shouldn't be `log.info` as that will be too noisy in the logs. If it indicates a bug, please bump it up to `log.warn`.

Target:
public static Expr binaryOp(BinaryOpExprBase binary, Expr left, Expr right) { try { return binary.getClass() .getDeclaredConstructor(String.class, Expr.class, Expr.class) .newInstance(binary.op, left, right); } catch (Exception e) { log.warn(e, "failed to rewrite expression " + binary); return binary; // best effort.. keep it working } }
Source:
private boolean possiblyStopAndRemoveSupervisorInternal(String id, boolean writeTombstone) { Pair<Supervisor, SupervisorSpec> pair = supervisors.get(id); if (pair != null) { if (writeTombstone) { metadataSupervisorManager.insert(id, new NoopSupervisorSpec()); // where NoopSupervisorSpec is a tombstone } pair.lhs.stop(true); START supervisors.remove(id); END return true; } return false; } I think you can move this to the top instead of the get.

Target:
private boolean possiblyStopAndRemoveSupervisorInternal(String id, boolean writeTombstone) { Pair<Supervisor, SupervisorSpec> pair = supervisors.get(id); if (pair == null) { return false; } if (writeTombstone) { metadataSupervisorManager.insert(id, new NoopSupervisorSpec()); // where NoopSupervisorSpec is a tombstone } pair.lhs.stop(true); supervisors.remove(id); return true; }
Source:
public boolean createOrUpdateAndStartSupervisor(SupervisorSpec spec) { START Preconditions.checkState(started, "SupervisorManager not started"); END Preconditions.checkNotNull(spec, "spec"); Preconditions.checkNotNull(spec.getId(), "spec.getId()"); synchronized (lock) { possiblyStopAndRemoveSupervisorInternal(spec.getId(), false); return createAndStartSupervisorInternal(spec, true); } } You'll want this inside the lock

Target:
public boolean createOrUpdateAndStartSupervisor(SupervisorSpec spec) { Preconditions.checkState(started, "SupervisorManager not started"); Preconditions.checkNotNull(spec, "spec"); Preconditions.checkNotNull(spec.getId(), "spec.getId()"); synchronized (lock) { Preconditions.checkState(started, "SupervisorManager not started"); possiblyStopAndRemoveSupervisorInternal(spec.getId(), false); return createAndStartSupervisorInternal(spec, true); } }
Source:
public boolean stopAndRemoveSupervisor(String id) { START Preconditions.checkState(started, "SupervisorManager not started"); END Preconditions.checkNotNull(id, "id"); synchronized (lock) { return possiblyStopAndRemoveSupervisorInternal(id, true); } } Same here, should be inside the lock

Target:
public boolean stopAndRemoveSupervisor(String id) { Preconditions.checkState(started, "SupervisorManager not started"); Preconditions.checkNotNull(id, "id"); synchronized (lock) { Preconditions.checkState(started, "SupervisorManager not started"); return possiblyStopAndRemoveSupervisorInternal(id, true); } }
Source:
START public CaffeineCache(final Cache<NamedKey, byte[]> cache, CaffeineCacheConfig config) END { this.cache = cache; this.config = config; } Looks like this could be private

Target:
private CaffeineCache(final Cache<NamedKey, byte[]> cache, CaffeineCacheConfig config) { this.cache = cache; this.config = config; }
Source:
public OffheapIncrementalIndex( long minTimestamp, QueryGranularity gran, final AggregatorFactory[] metrics, boolean deserializeComplexMetrics, boolean reportParseExceptions, boolean sortFacts, int maxRowCount, StupidPool<ByteBuffer> bufferPool ) { this( new IncrementalIndexSchema.Builder().withMinTimestamp(minTimestamp) .withQueryGranularity(gran) .withMetrics(metrics) START .withRollup(DEFAULT_ROLLUP) END .build(), deserializeComplexMetrics, reportParseExceptions, sortFacts, maxRowCount, bufferPool ); } can this use incrementalIndexSchema.isRollup?

Target:
public OffheapIncrementalIndex( long minTimestamp, QueryGranularity gran, final AggregatorFactory[] metrics, boolean deserializeComplexMetrics, boolean reportParseExceptions, boolean sortFacts, int maxRowCount, StupidPool<ByteBuffer> bufferPool ) { this( new IncrementalIndexSchema.Builder().withMinTimestamp(minTimestamp) .withQueryGranularity(gran) .withMetrics(metrics) .withRollup(IncrementalIndexSchema.DEFAULT_ROLLUP) .build(), deserializeComplexMetrics, reportParseExceptions, sortFacts, maxRowCount, bufferPool ); }
Source:
public boolean supportsBitmapIndex(BitmapIndexSelector selector) { START if(selector.getBitmapIndex(boundDimFilter.getDimension()) == null) { END return false; } else { return true; } } formatting

Target:
public boolean supportsBitmapIndex(BitmapIndexSelector selector) { return selector.getBitmapIndex(boundDimFilter.getDimension()) != null; }
Source:
public boolean supportsBitmapIndex(BitmapIndexSelector selector) { START if(selector.getBitmapIndex(dimension) == null) { END return false; } else { return true; } } formatting

Target:
public boolean supportsBitmapIndex(BitmapIndexSelector selector) { return selector.getBitmapIndex(dimension) != null; }
Source:
RetryPolicyFactory createRetryPolicyFactory() { START return new RetryPolicyFactory(new RetryPolicyConfig() END .setMinWait(Period.seconds(2)) .setMaxWait(Period.seconds(8)) .setMaxRetryCount(5) ); } Since @dclim is also the original author if these numbers, that would be nice.

Target:
RetryPolicyFactory createRetryPolicyFactory() { // Retries for about a minute before giving up; this should be long enough to handle any temporary unresponsiveness // such as network issues, if a task is still in the process of starting up, or if the task is in the middle of // persisting to disk and doesn't respond immediately. return new RetryPolicyFactory( new RetryPolicyConfig() .setMinWait(Period.seconds(2)) .setMaxWait(Period.seconds(8)) .setMaxRetryCount(8) ); }
Source:
public boolean aggregate(KeyType key) { final int keyHash = Groupers.hash(key); START final int grouperNumber = grouperNumberForKeyHash(keyHash); END final Grouper<KeyType> grouper = groupers.get(grouperNumber); synchronized (grouper) { return grouper.aggregate(key, keyHash); } } can do return aggregate(key, keyHash)

Target:
public boolean aggregate(KeyType key) { return aggregate(key, Groupers.hash(key)); }
Source:
private LookupExtractionFn ensureDelegate() { START LookupExtractionFn delegate = this.delegate; END if (null == delegate) { // http://www.javamex.com/tutorials/double_checked_locking.shtml synchronized (delegateLock) { delegate = this.delegate; if (null == delegate) { delegate = new LookupExtractionFn( Preconditions.checkNotNull(manager.get(getLookup()), "Lookup [%s] not found", getLookup()).get(), isRetainMissingValue(), getReplaceMissingValueWith(), isInjective(), isOptimize() ); this.delegate = delegate; } } } return delegate; } why not always use this.delegate directly and avoid the extra local variable? it would make things shorter and less confusing.

Target:
private LookupExtractionFn ensureDelegate() { if (null == delegate) { // http://www.javamex.com/tutorials/double_checked_locking.shtml synchronized (delegateLock) { if (null == delegate) { delegate = new LookupExtractionFn( Preconditions.checkNotNull(manager.get(getLookup()), "Lookup [%s] not found", getLookup()).get(), isRetainMissingValue(), getReplaceMissingValueWith(), isInjective(), isOptimize() ); } } } return delegate; }
Source:
public int hashCode() { START int result = container != null ? container.hashCode() : 0; END result = 31 * result + (aggregators != null ? Arrays.hashCode(aggregators) : 0); result = 31 * result + (queryGranularity != null ? queryGranularity.hashCode() : 0); return result; } `container` is non-null

Target:
public int hashCode() { int result = container.hashCode(); result = 31 * result + (aggregators != null ? Arrays.hashCode(aggregators) : 0); result = 31 * result + (queryGranularity != null ? queryGranularity.hashCode() : 0); return result; }
Source:
public ByteBuffer makeNewBuffer(long offset) { if (sizeAvailable.addAndGet(-1 * chunkSize) > 0) { return delegate.makeNewBuffer(offset); } else { START throw new RuntimeException("No more space left for allocating buffers."); END } } this will decrease the availableSize and throw exception. this can possibly cause availableSize to get out of sync from the actual allocated buffer.

Target:
public ByteBuffer makeNewBuffer(long offset) { if (sizeAvailable.addAndGet(-1 * chunkSize) > 0) { return delegate.makeNewBuffer(offset); } else { sizeAvailable.addAndGet(chunkSize); throw new RuntimeException("No more space left for allocating buffers."); } }
Source:
public V getIfPresent(K key) { V value = cache.get(key); if (value == null) { START missCount.incrementAndGet(); END } else { hitCount.getAndIncrement(); } return value; } For readability sake, can these two increments use the same atomic method call?

Target:
public V getIfPresent(K key) { V value = cache.get(key); if (value == null) { missCount.getAndIncrement(); } else { hitCount.getAndIncrement(); } return value; }
Source:
public V get(K key, final Callable<? extends V> valueLoader) throws ExecutionException { V value = cache.get(key); if (value != null) return value; try { value = valueLoader.call(); START cache.put(key, value); END return value; } catch (Exception e) { throw new ISE(e, "got an exception while loading key [%s]", key); } } not thread safe

Target:
public V get(K key, final Callable<? extends V> valueLoader) throws ExecutionException { synchronized (key) { V value = cache.get(key); if (value != null) { return value; } try { value = valueLoader.call(); cache.put(key, value); return value; } catch (Exception e) { throw new ISE(e, "got an exception while loading key [%s]", key); } } }
Source:
public List<? extends Module> getJacksonModules() { return ImmutableList.<Module>of( new SimpleModule("DruidNamespacedExtractionModule") START .registerSubtypes(NamespacedExtractor.class) END .registerSubtypes(ExtractionNamespace.class) .registerSubtypes(NamespaceLookupExtractorFactory.class) ); } these can all be on the same method call

Target:
public List<? extends Module> getJacksonModules() { return ImmutableList.<Module>of( new SimpleModule("DruidNamespacedExtractionModule") .registerSubtypes( ExtractionNamespace.class, NamespaceLookupExtractorFactory.class ) ); }
Source:
public static List<Filter> toFilters(List<DimFilter> dimFilters) { return ImmutableList.copyOf( FunctionalIterable .create(dimFilters) START .filter( END new Predicate<DimFilter>() { @Override public boolean apply(DimFilter input) { return input != null; } } ).transform( new Function<DimFilter, Filter>() { @Override public Filter apply(DimFilter input) { return input.toFilter(); } } ) ); } Should we throw an exception instead if a DimFilter was null?

Target:
public static List<Filter> toFilters(List<DimFilter> dimFilters) { return ImmutableList.copyOf( FunctionalIterable .create(dimFilters) .transform( new Function<DimFilter, Filter>() { @Override public Filter apply(DimFilter input) { return input.toFilter(); } } ) ); }
Source:
public SelectorFilter( String dimension, String value, ExtractionFn extractionFn ) { this.dimension = dimension; START this.value = Strings.nullToEmpty(value);; END this.extractionFn = extractionFn; } double `;;`

Target:
public SelectorFilter( String dimension, String value, ExtractionFn extractionFn ) { this.dimension = dimension; this.value = Strings.nullToEmpty(value); this.extractionFn = extractionFn; }
Source:
public ValueMatcher makeMatcher(ValueMatcherFactory factory) { if (extractionFn == null) { return factory.makeValueMatcher(dimension, value); } else { return factory.makeValueMatcher( dimension, new Predicate<String>() { @Override public boolean apply(String input) { // Assuming that a null/absent/empty dimension are equivalent from the druid perspective START return value.equals(Strings.nullToEmpty(extractionFn.apply(Strings.emptyToNull(input)))); END } } ); } } I don't think the `Strings.emptyToNull` is needed on the input to the extractionFn; `input` shouldn't be empty anyway (ValueMatcherFactories aren't passing in empty strings to the predicate) and also most other callers of extractionFns don't do this.

Target:
public ValueMatcher makeMatcher(ValueMatcherFactory factory) { if (extractionFn == null) { return factory.makeValueMatcher(dimension, value); } else { return factory.makeValueMatcher( dimension, new Predicate<String>() { @Override public boolean apply(String input) { // Assuming that a null/absent/empty dimension are equivalent from the druid perspective return value.equals(Strings.nullToEmpty(extractionFn.apply(input))); } } ); } }
Source:
public AsyncQueryForwardingServlet( @Json ObjectMapper jsonMapper, @Smile ObjectMapper smileMapper, QueryHostFinder hostFinder, @Router Provider<HttpClient> httpClientProvider, DruidHttpClientConfig httpClientConfig, ServiceEmitter emitter, RequestLogger requestLogger ) { this.jsonMapper = jsonMapper; this.smileMapper = smileMapper; this.hostFinder = hostFinder; this.httpClientProvider = httpClientProvider; this.httpClientConfig = httpClientConfig; this.emitter = emitter; this.requestLogger = requestLogger; // separate client with more aggressive connection timeouts // to prevent cancellations requests from blocking queries this.broadcastClient = httpClientProvider.get(); broadcastClient.setConnectTimeout(CANCELLATION_TIMEOUT); START broadcastClient.setMaxRequestsQueuedPerDestination(MAX_QUEUED_CANCELLATIONS); END } should we also lower down maxConnectionsPerDestination ?

Target:
public AsyncQueryForwardingServlet( @Json ObjectMapper jsonMapper, @Smile ObjectMapper smileMapper, QueryHostFinder hostFinder, @Router Provider<HttpClient> httpClientProvider, DruidHttpClientConfig httpClientConfig, ServiceEmitter emitter, RequestLogger requestLogger ) { this.jsonMapper = jsonMapper; this.smileMapper = smileMapper; this.hostFinder = hostFinder; this.httpClientProvider = httpClientProvider; this.httpClientConfig = httpClientConfig; this.emitter = emitter; this.requestLogger = requestLogger; }
Source:
private DatasourceInputSplit toDataSourceSplit( List<WindowedDataSegment> segments, org.apache.hadoop.mapred.InputFormat fio, JobConf conf ) { String[] locations = null; try { locations = getFrequentLocations(segments, fio, conf); } catch (Exception e) { logger.error(e, "Exception thrown finding location of splits"); START throw Throwables.propagate(e); END } return new DatasourceInputSplit(segments, locations); } i don't this needs to be propagated. finding locations should only be best effort.

Target:
private DatasourceInputSplit toDataSourceSplit( List<WindowedDataSegment> segments, org.apache.hadoop.mapred.InputFormat fio, JobConf conf ) { String[] locations = null; try { locations = getFrequentLocations(segments, fio, conf); } catch (Exception e) { logger.error(e, "Exception thrown finding location of splits"); } return new DatasourceInputSplit(segments, locations); }
Source:
START public Set<String> getRunningTaskIds() END { return getRunningTasks().keySet(); } Does this need to be Set?

Target:
public Collection<String> getRunningTaskIds() { return getRunningTasks().keySet(); }
Source:
public List<String> getPartitionDimensions() { if (partitionDimension != null) { return ImmutableList.of(partitionDimension); } else { START return null; END } } can you add `@Nullable` as method signature

Target:
public List<String> getPartitionDimensions() { return ImmutableList.of(); }
Source:
public void setUp() throws Exception { amazonEC2Client = EasyMock.createMock(AmazonEC2Client.class); START mockAmazonEC2Client = new EC2MockClient(); END describeInstancesResult = EasyMock.createMock(DescribeInstancesResult.class); reservation = EasyMock.createMock(Reservation.class); instance = new Instance() .withInstanceId(INSTANCE_ID) .withLaunchTime(new Date()) .withImageId(AMI_ID) .withPrivateIpAddress(IP); managementConfig = new SimpleResourceManagementConfig().setWorkerPort(8080).setWorkerVersion(""); } can you use EasyMock instead of creating a new class ? It doesn't seem to do anything custom else except get/set region.

Target:
public void setUp() throws Exception { amazonEC2Client = EasyMock.createMock(AmazonEC2Client.class); describeInstancesResult = EasyMock.createMock(DescribeInstancesResult.class); reservation = EasyMock.createMock(Reservation.class); instance = new Instance() .withInstanceId(INSTANCE_ID) .withLaunchTime(new Date()) .withImageId(AMI_ID) .withPrivateIpAddress(IP); managementConfig = new SimpleResourceManagementConfig().setWorkerPort(8080).setWorkerVersion(""); }
Source:
public Iterator<String> iterator() { START LOG.debug("Not json, hoping it is a good string : %s", string); END return Splitter.on( new CharMatcher() { private boolean inQuotes = false; @Override public boolean matches(char c) { if ('"' == c) { inQuotes = !inQuotes; } if (inQuotes) { return false; } return CharMatcher.BREAKING_WHITESPACE.matches(c); } } ).omitEmptyStrings().split(string).iterator(); } this debug msg is not valid anymore.

Target:
public Iterator<String> iterator() { return Splitter.on( new CharMatcher() { private boolean inQuotes = false; @Override public boolean matches(char c) { if ('"' == c) { inQuotes = !inQuotes; } if (inQuotes) { return false; } return CharMatcher.BREAKING_WHITESPACE.matches(c); } } ).omitEmptyStrings().split(string).iterator(); }
Source:
public void testBootstrapFromFile() throws IOException { START LookupExtractorFactory lookupExtractorFactory = new LookupExtractorFactoryMock("data"); END mapper.registerSubtypes(LookupExtractorFactoryMock.class); lookupReferencesManager.put("testMockForBootstrap",lookupExtractorFactory); lookupReferencesManager.stop(); lookupReferencesManager.start(); Assert.assertEquals(lookupExtractorFactory, lookupReferencesManager.get("testMockForBootstrap")); } Can this simply be an EasyMock?

Target:
public void testBootstrapFromFile() throws IOException { LookupExtractorFactory lookupExtractorFactory = new LookupExtractorFactoryMock("data"); lookupReferencesManager.put("testMockForBootstrap",lookupExtractorFactory); lookupReferencesManager.stop(); lookupReferencesManager.start(); Assert.assertEquals(lookupExtractorFactory, lookupReferencesManager.get("testMockForBootstrap")); }
Source:
START public FireHydrant getCurrHydrant() END { synchronized (hydrantLock) { return currHydrant; } } `@VisibleForTesting`, could also be package-private.

Target:
FireHydrant getCurrHydrant() { synchronized (hydrantLock) { return currHydrant; } }
Source:
START public void testSerde() throws JsonParseException, JsonMappingException, JsonProcessingException, IOException END { ObjectMapper mapper = new DefaultObjectMapper(); SketchEstimateWithErrorBounds est = new SketchEstimateWithErrorBounds(100.0,101.5,98.5,2); Assert.assertEquals(est, mapper.readValue( mapper.writeValueAsString(est), SketchEstimateWithErrorBounds.class)); } need only `IOException`

Target:
public void testSerde() throws IOException { ObjectMapper mapper = new DefaultObjectMapper(); SketchEstimateWithErrorBounds est = new SketchEstimateWithErrorBounds(100.0,101.5,98.5,2); Assert.assertEquals(est, mapper.readValue( mapper.writeValueAsString(est), SketchEstimateWithErrorBounds.class)); }
Source:
private ValueType getTypeFromDimVal(Object dimVal) { Object singleVal; if (dimVal instanceof List) { List dimValList = (List) dimVal; singleVal = dimValList.size() == 0 ? null : dimValList.get(0); } else { singleVal = dimVal; } if (singleVal == null) { return null; } START ValueType valType = TYPE_MAP.get(singleVal.getClass().getSimpleName()); END return valType; } can directly return without storing in `valType` local variable

Target:
private ValueType getTypeFromDimVal(Object dimVal) { Object singleVal; if (dimVal instanceof List) { List dimValList = (List) dimVal; singleVal = dimValList.size() == 0 ? null : dimValList.get(0); } else { singleVal = dimVal; } if (singleVal == null) { return null; } return TYPE_MAP.get(singleVal.getClass().getSimpleName()); }
Source:
START protected Collection<Worker> getWorkerFromZK(Collection<ZkWorker> workers) END { return Collections2.transform( workers, new Function<ZkWorker, Worker>() { @Override public Worker apply(ZkWorker input) { return input.getWorker(); } } ); } could this be `private`? RTR doesn't have subclasses as far as I know

Target:
public static Collection<Worker> getWorkerFromZK(Collection<ZkWorker> workers) { return Collections2.transform( workers, new Function<ZkWorker, Worker>() { @Override public Worker apply(ZkWorker input) { return input.getWorker(); } } ); }
Source:
public ExtractionType getExtractionType(){ START return (child != null) END ? ((child.getExtractionType() == ExtractionType.MANY_TO_ONE) ? ExtractionType.MANY_TO_ONE : fn.getExtractionType() ) : fn.getExtractionType(); } I think this set of conditions is a bit hard to read, can you change it to something like: if (child != null && child.getExtractionType() == ExtractionType.MANY_TO_ONE) { return ExtractionType.MANY_TO_ONE; } else { return fn.getExtractionType(); }

Target:
public ExtractionType getExtractionType(){ if (child != null && child.getExtractionType() == ExtractionType.MANY_TO_ONE) { return ExtractionType.MANY_TO_ONE; } else { return fn.getExtractionType(); } }
Source:
public String toString() { StringBuilder sb = new StringBuilder(); sb.append("DimensionSelectorHavingSpec"); sb.append("{dimension='").append(dimension).append('\''); sb.append(", value='").append(value); START if (getExtractionFn() != null) { END sb.append("', extractionFunction='").append(getExtractionFn()); } sb.append("'}"); return sb.toString(); } extractionfn is never null

Target:
public String toString() { StringBuilder sb = new StringBuilder(); sb.append("DimensionSelectorHavingSpec"); sb.append("{dimension='").append(dimension).append('\''); sb.append(", value='").append(value); sb.append("', extractionFunction='").append(getExtractionFn()); sb.append("'}"); return sb.toString(); }
Source:
private static Set<String> getWorkerNodeIDs(Collection<Worker> workers, WorkerBehaviorConfig workerConfig) { START return Sets.<String>newHashSet(workerConfig.getAutoScaler().ipToIdLookup( END Lists.newArrayList( Iterables.transform( workers, new Function<Worker, String>() { @Override public String apply(Worker input) { return input.getIp(); } } ) ) )); } creating a set might not always be necessary, a collection should be enough for all the cases where we use getWorkerNodeIDs

Target:
private static Collection<String> getWorkerNodeIDs(Collection<Worker> workers, WorkerBehaviorConfig workerConfig) { return workerConfig.getAutoScaler().ipToIdLookup( Lists.newArrayList( Iterables.transform( workers, new Function<Worker, String>() { @Override public String apply(Worker input) { return input.getIp(); } } ) ) ); }
Source:
START public static List<String> readStringArray(DataInput in) throws IOException END { int count = WritableUtils.readVInt(in); if(count == 0) { return null; } List<String> values = Lists.newArrayListWithCapacity(count); for(int i = 0; i < count; i++) { values.add(readString(in)); } return values; } all these util methods should be private.

Target:
private static List<String> readStringArray(DataInput in) throws IOException { int count = WritableUtils.readVInt(in); if (count == 0) { return null; } List<String> values = Lists.newArrayListWithCapacity(count); for (int i = 0; i < count; i++) { values.add(readString(in)); } return values; }
Source:
public void testQuerySerialization() throws IOException { RegexSearchQuerySpec spec = new RegexSearchQuerySpec("(upfront|total_market)"); String json = jsonMapper.writeValueAsString(spec); START RegexSearchQuerySpec serdeQuery = jsonMapper.readValue(json, RegexSearchQuerySpec.class); END Assert.assertEquals(spec, serdeQuery); } can u change to ``` (RegexSearchQuerySpec) jsonMapper.readValue(json, SearchQuerySpec.class); ```

Target:
public void testQuerySerialization() throws IOException { RegexSearchQuerySpec spec = new RegexSearchQuerySpec("(upfront|total_market)"); String json = jsonMapper.writeValueAsString(spec); RegexSearchQuerySpec serdeQuery = (RegexSearchQuerySpec) jsonMapper.readValue(json, RegexSearchQuerySpec.class); Assert.assertEquals(spec, serdeQuery); }