Issue 1:
With the rename to argmax, and dims now a kwarg, it seems very tempting to me to define a 2-arg flavor, argmax(f, itr), where itr is an iterable subset of the domain of f. This would return an x in itr that maximizes f(x), which is quite literally the mathematical definition of argmax.

(originally proposed here)

Edit: Previous wording may have been confusing, so I have attempted to improve it.

Issue 2:
As progress moves forward on v0.5 development (especially #13157), we'll need an automated system for executing benchmarks and identifying performance regressions.

The Julia group has recently purchased dedicated performance testing hardware which is hosted at CSAIL, and I've been brought on to facilitate the development of a system that takes advantage of this hardware. I'm hoping we can use this issue to centralize discussion/development efforts.

Desired features

Any implementation of a performance tracking system should:

track multiple metrics (e.g. time, % GC time, bytes allocated)

allow collaborators to trigger the following via GitHub's UI:

benchmark execution on a specific commit

benchmark result comparison between two specific commits

incorporate a tagging system so that groups of benchmarks can be selectively run by topic (e.g. run only parallel benchmarks, or only string benchmarks)

store benchmark result history for future analysis

report benchmark status/results through GitHub's UI

be applicable to normal Julia packages, not just Base (envision a future where PackageEvaluator benchmarks registered packages)

preferably be written in native Julia

Feel free to chime in with additional feature goals - the ones listed above just outline what I've been focusing on so far.

Existing work

In order to make progress on this issue, I've been working on JuliaCI/BenchmarkTrackers.jl, which supplies a unified framework for writing, executing, and tracking benchmarks. It's still very much in development, but currently supports all of the goals listed above. I encourage you to check it out and open up issues/PRs in that repository if you have ideas or concerns. Just don't expect the package to be stable yet - I'm in the process of making some drastic changes (mainly to improve the testability of the code).

Here are some other packages that any interested parties will want to be familiar with:

johnmyleswhite/Benchmarks.jl: a framework for benchmark execution, with the goal of delivering more rigorous/useful metrics than those provided by simple calls to @time.

JuliaWeb/GitHub.jl: will soon contain the GitHub event-handling infrastructure that I've been working on, which allows stuff like comment-based triggers and GitHub status reporting.

IainNZ/BasePerfTests.jl: a collection of performance tests (I'm not sure that this is still being developed).

staticfloat/Perftests.jl: another collection of performance tests + a grouping/logging framework for ease of execution. I believe @staticfloat has been running these on occasion via the buildbots.

Eventually we will want to consolidate "blessed" benchmarking/CI packages under an umbrella Julia group on Github (maybe JuliaCI)?

I saw that Codespeed was used for a while, but that effort was abandoned due to the burden of maintaining the server through volunteer effort. I've also been told that Codespeed didn't integrate well with the Github-centric CI workflow that we've become accustomed to.

Resolving this issue

Taking into account the capabilities previously mentioned, I imagine that a single CI benchmark cycle for Base would go through these steps:

Collaborator comments on a commit with the appropriate trigger phrase

A comment-listening server (the meat of which I've implemented here) triggers the buildbots a CI server to build Julia at that commit

The buildbots call out to BenchmarkTrackers is used to process an external package of performance tests (similar to Perftests, but written using BenchmarkTrackers)

BenchmarkTrackers reports the results via GitHub statuses to the commit. Any additional reporting can be done with the @nanosoldier bot ("nanosolider" is the hostname of the hardware at CSAIL, and @jakebolewski managed to grab the GitHub account name).

If we can deliver the workflow described above, I think that will be sufficient to declare this issue "resolved."

Next steps

The following still needs to get done:

Configure a webhook on the JuliaLang/julia repo for firing off events so that we can begin testing the necessary CI infrastructure

Get BenchmarkTrackers.jl to a release-ready state, which means more robust testing and further development on both itself and its dependencies, GitHub.jl and Benchmarks.jl

Start organizing/rewriting existing performance tests using BenchmarkTrackers.jl

Set up the @nanosoldier to do any additional status reporting we want (e.g. linking to a commit status page after somebody triggers benchmark execution - GitHub doesn't do a good job of displaying statuses for commits outside of PRs).

Regression Examples

Here are some examples from regression-prone areas that I think could be more easily moderated with an automated performance tracking system:

parallel (#12794, #12223)

inlining/type inference (#13551, #13350, #12476)

"real-world" use cases (#11823, #7000, #6415)

individual functions (#11700, #10650, #8100, #6112)

codegen/vectorization (#13777, #13301, #11899, #11997)