Research guidelines in Deep Learning
Jan van Gemert
Delft University of Technology
http://jvgemert.github.io/

? ??
!
??
!!
?
!! !
1

Abstract
The goal of this document is to give structure to doing deep learning research. It might be useful for students, advisors, researchers,or as part of a (research) course. The beauty of research is that it’screative, inspiring, and different every time. Yet, over the years, I’ve
found myself repeating different variants of the same feedback multiple times. This document contains a set of 150 loose guidelines,
structured as uniquely named-item checklists which aims to structure common research settings and I find them useful as alignment,
as a time saver during meetings, as a feedback tool, or rubric. These
guidelines revolve around embracing the chaotic, creative, exploratory
search, where it is completely normal to revisit and re-evaluate everything along the way. This chaotic research process goes through highs
and lows which can be exciting and inspiring, and at the same time
can be scary and demotivating. This is inherent to creative processes
and also in doing research; and the key is to stubbornly keep the faith!
Something will come and these guidelines help find it. They focus onunderstanding-based empirical deep learning research, without necessarily having access to huge compute/data. I believe that criticallyrethinking the foundations of deep learning models will foster exciting
creative avenues and revolutionary different research directions. Better understanding the fundaments will lead to safer, transparent, morealigned, less data/compute dependent, and more robust AI systems.

2

Contents
1 Why these guidelines?

4

2 The Storyline

8

3 Research Organization, Research Process, and Research Mentality
94 Research meeting questions

14

5 Writing guidelines

19

6 Giving a Talk

28

7 Poster guidelines

31

8 Reviewing and rebuttal guidelines

34

9 Appendix: Separately printable tables.

39

3

1

Why these guidelines?

The goal of this document is to give structure to doing deep learning research. It might be useful for students, advisors, researchers, or as part ofa (research) course. The beauty of research is that it’s creative, inspiring,and different every time. Different people do research in different ways, andresearchers/advisors may not agree with all my guidelines, which, if properly motivated, is perfectly fine. This document is thus not intended as afixed rigid protocol [3]. Instead, it is a set of 150 loose guidelines, structured as named-item checklists that I have found repeatedly useful over the
many years that I have been doing research that I decided to organize anddocument them. I’ve found it useful as alignment, as a time saver during
meetings, as a feedback tool, and as a reminder to myself; I can even see it
useful as a point of reference in doing paper reviews and rebuttals.
These guidelines can be characterized as understanding-based empiricaldeep learning research. It’s emprical instead of theoretical, and thus focusesless on mathematical proofs and more on data-driven empirical results. It’s
understanding-based because I believe the goal of science is understanding,even (especially?) in the fast-moving field of Deep Learning. I believe thatcritically rethinking the foundations of deep learning models will foster exciting creative avenues and revolutionary different research directions. Better
understanding the fundaments will lead to safer, transparent, more aligned,less data/compute dependent, and more robust AI systems.
Regarding scope, I certainly do not aim to claim that deep learning is
the only relevant machine learning setting, it’s just the paradigm in my own
research field. This field is huge, and these guidelines might not apply to
all deep learning research projects; while at the same time several guidelinesalign well with a broader research domain than deep learning. If you foundthese guidelines useful in your scientific approach or otherwise, then pleasefeel very free to say why, and cite these guidelines in your scientific paper,
thesis, or project.

1.1

Other research guidelines

I do not wish to start with the philosophy of science, where one can argue
if Deep Learning is a paradigm shift as identified by Kuhn’s Structure of
Scientific Revolutions [5]. I agree with Feyerabend’s Anarchistic Theory ofKnowledge [3] and I do not subscribe to a strict protocol to do science. Yet, I4

do see that there are common repeating patterns that can serve as guidelines.
In the field of computer science there are guidelines [1], and also in machinelearning [2], which are useful and complementary. In this document I go
deeper and identify over 150 uniquely named and referable guidelines spreadout over 7 sections.
My guidelines focus on understanding-based empirical deep learning research. The advantages of such an approach are excellently demonstrated in
the “Scaling down Deep Learning” paper [4]. Several of my guidelines emphasize critical thinking, where a special mention goes to the “Troubling Trends”
paper [6], which is an excellent manifesto of things that can be improved,
and all students in my Deep Learning MSc course have read it. On a similarnote, albeit more targeted at academic researchers, there are the “Survival
Strategies for Depressed AI Academics” [7] which argues for keep doing Deep
Learning research, even as a poor academic, without having access to hugedata and compute resources. On this note, I often remind students that it
wasn’t the industry who kept the field of Deep Learning alive before 2012:
Deep Learning was studied as a niche, by poor academics, at a university.
There are many important unsolved research topics in Deep Learning, including fairness, alignment, robustness, transparency, explainability and thedependency on large data/compute. This brings me to the following quoteby Geoffrey Hinton: “The future depends on some graduate student who is
deeply suspicious of everything I have said.”. We need to keep doing critical,
fundamental Deep Learning research in academia; question everything, go
against the trend, the next revolution is waiting for you!

1.2

Approach to Deep Learning research

Research is risky These guidelines revolve around embracing the chaotic,
creative, exploratory search in finding the research question, the work related
to it, a method or approach to address the research question, the empiricalquestions to answer in the experiments, the analysis, and conclusions. It iscompletely normal to revisit and re-evaluate everything along the way, as
illustrated in Fig 1. Research is inherently risky, and the research processthus is to a large part about risk management. So, do the most risky partfirst. Don’t invest time in doing the ’easy’ things that you expect to work out
because it might be the case that these things are no longer relevant later.
Put the risk first; try to break the idea as soon as possible.

5CommunicatedActual process

Research
Question

Relatedwork

Research
Question

Relatedwork

Drawing
conclusions

Method/ApproachDrawing
conclusions

Method/Approach

Analyzing
results

Experimental
setup

Analyzing
results

Experimental
setup

Figure 1: How deep learning research is typically communicated vs an example of how the actual scientific process can go (red line). It is completely
normal during the research process to rephrase the research question often;
to revisit related work in a new light, to change the method or experimental
setup, and re-analyze results and conclusions.
Find the path by taking a step Often, there is some direction, topic, or
scope, but no clear well-defined end-goal when starting the research process.
Starting this process fully aware that it will be chaotic, noisy, and go throughhighs and lows can be exciting and inspiring, and at the same time can bescary and demotivating. This is inherent to creative processes and also indoing research; and the key is to stubbornly keep the faith! Somethinginteresting will come out. Yet, before being able to find the route, there has
to be some exploration. It’s easy by ’armchair philosophy’ to dismiss possible
directions or refuse to setup a simple fully controlled version of the problem
by reasoning away that it’s trivial. To arrive at a well-defined end-goal, there
has to be a first step, even if it might be in the ’wrong’ direction.
Empirical research Experiments are empirical support for the hypothesis, claims, observations, and benefits. I typically identify 3 general experiment types, where each type can have multiple experimental settings.
Type 1: Validate claims. Does the problem exist? Validate that baselines
suffer from the identified problem. Does your proposed method address the
problem? Etc. This is often best done in a fully-controlled (toy) experimental
setting where having full control allows validating the problem and demonstrating the solution. Type 2: variations. Report interesting variations andexplain why they are interesting. Do ablation studies on individual components. Show sensitivity to hyper-parameters. Etc. Type 3: Existing data sets
Does the identified problem occur in existing less controlled datasets? Is6the problem common? Can you show that the problem occurs in multiple
datasets? Validate that baselines still suffer from the problem. Verify thatthe proposed method addresses the problem. You are free to choose any
existing dataset; so choose datasets that match your problem.

1.3

How this document is structured

This document is a collection of 150 guidelines spread over several sections:
Section 2: The Storyline.
Section 3: Organization, process, and mentality.
Section 4: Research meetings.
Section 5: Writing.
Section 6: Giving a talk.
Section 7: Presenting a poster.
Section 8: Reviewing and rebuttals.
For relevant sections, I added a table with each guideline/question uniquelylabeled; which can be used to give feedback, to point at during a meeting, or
to use as a reminder. In the appendix I copied all tables on a separate pagefor convenient printing and adding your own.

7

2

The Storyline

The storyline is one of the most important tools in research: It helps tostructure several things: thoughts, meetings, the process, the research questions, and enormously helps the writing. It is normal that the storylinechanges/evolves many times, see Fig 1.
This is the typical structure of a storyline; each element might take 1-3
bullet points:(1). Why interesting? What is the general setting/application. Why
should people care (not researchers; but the users of the research outcome).
(2). How done now? What are the typical methods/approach(es) tothe setting in (1)(3). What is missing? What’s the problem with the approaches in (2),and what consequences does this have.
(4). Proposed solution. What do you do, and why does it solve theproblem in (3)?
(5). Experimental questions. How do you evaluate experimentallythat (4) solves the problem in (2) and it’s consequences in (3).
The storyline is stand alone: you cannot introduce a new term/concept
before it has been introduced by motivating it; ie: what is it and why is
it needed. Keep Hitchens’s razor in mind: “What can be asserted without
evidence can also be dismissed without evidence. Terms/concepts logicallybuild/connect to earlier terms/concepts (ie: its a story). Have short “1liners” per bullet point; correct grammar is optional; the entire storyline
should be visible at once, so it should fit on half a page or a single slide(10-15 bullet points).

8

3

Research Organization, Research Process,and Research Mentality

Doing research often assumes a certain mindset, process and organization.
This holds for the individual researcher, but also for an advisor. Here, I makesuch assumptions explicit.

3.1

Organization

Process

Mentality

RO1 Full responsibility
RO2 No dependencies
RO3 Meet advisor
RO4 Focus advisor
RO5 Take critique
RO6 Constructive disagree
RO7 Analyze results
RO8 Suggest solutions
RO9 Give feedback
RO10 Safety

RP1 One main Q
RP2 Min. 3rd party
RP3 Validate baselines
RP4 First break it
RP5 Depth first
RP6 Exps answer Q
RP7 Proof of concept
RP8 Exps max 1 night
RP9 Change 1 var
RP10 Debug science
RP11 Figures

RM1 Be critical
RM2 Find todos together
RM3 Consistency
RM4 Question everything
RM5 Simple is strong
RM6 Embrace limitations
RM7 Write early and often
RM8 Not eureka
RM9 Show the problem
RM10 Motivate everything

Research Organization

RO1: Take full responsibility. This is your project. Not your adviser’s.
You are in charge of everything, including: planning, progress, direction,
meeting topics, bureaucratic formalities, etc. You are not alone: youradviser is there to help you as best as possible, yet the final responsibility
remains yours.
RO2: No dependencies. Avoid dependencies on third parties. Such
parties intend well, yet reality is often different from intentions. Do not
become the victim of this and make sure you have full control. E.g.:
Promised data, labels, experts, constraints, or other agreements have to be
there before you start.
RO3: Meet your adviser. Try to see your adviser at least once every 2
weeks; once per week is better.

9

RO4: Focus your adviser. Meeting time is limited. Avoid needlesschronological updates (no need for “proof of work”). Discuss problems,
choices, dilemmas and directions. It is your responsibility to choose to
discuss what benefits you most.
RO5: Do not take criticism personal. All feedback is meant toimprove and benefit you. Do not fight the feedback, even if you think it
is wrong: Make a note, and think about why your advisor gave this feedback.
RO6: Constructive disagree. It’s OK to disagree with a suggestionof your adviser, but if you repeatedly do this then also try to propose
something yourself.
RO7: Analyze results. When presenting (intermediate) results, give aninterpretation and conclusions (ie: answer the “So what?” question).
RO8: Suggest solutions. When encountering research or organizational
problems; suggest a solution yourself.
RO9: Give feedback to your advisor If you are unhappy aboutsomething (eg: how feedback is given, how meetings go, meeting time; etc.)
then please let your advisor know this. Your advisor cannot read your mind,
and is there to help you, give your advisor the opportunity to help you best.
RO10: Safe environment Meetings and the advisor-advisee relationshipshould be safe and based on mutual respect. If you do not feel safe, contacta (confidential) counselor at your organization, your advisor’s advisor, andspeak out to friends/peers. It is your advisor that needs to change.

3.2

The research process

RP1: Only one main research question / problem statement. It
may change over time but explicitly pursue a single topic: Write it down tomake it precise; this gives focus and direction.

10

RP2: Minimal effort for 3rd party building blocks. If you build on
top of existing work (e.g. an optimizer, object detector, pose estimator, etc.)
start with the least effort approach to obtain this building block. It should
not matter which building block you take, so start with the easiest available
implementation. If you work modularity, you can always add another one
later.
RP3: Validate published work. It is not obvious that a publishedwork generalizes to your problem. There may be subtleties: Validate this.
RP4: Prioritize idea breaking. Start by investigating the greatest risk
to your main research question. Do not invest heavily on the foundations,only to find out months later that the main idea did not work.
RP5: Depth-first instead of breadth-first. Do not explore sub-topics
too deep. Identify the minimum requirement per sub-topics and get to thisminimum as soon as possible. Try to get ASAP to a first full version tovalidate your idea. More baselines/variants/datasets can always be done
later.
RP6: Experiments answer a single question.
Write down before
you do an experiment what your expected answer to the question is. Validate.
RP7: Show proof of concept. Start with a fully controlled (possiblytoy) dataset of ’the simplest case possible’ which should only vary in therelevant manner. Its goal is to validate that the problem occurs and/or thatyour model can solve it.
RP8: Experiments take max 1 night. If it takes 1 week, then 10
runs take 2.5 months. Minimize experimental time so you can answer morequestions, especially in the beginning; leave larger experiments for the end.
RP9: Change only one variable. If more than one variable is changed,
it is not possible to determine the cause of an effect.
RP10: Debug your scientific ideas and your code. Test ideas andtest code every time you make a change. Start with the assumption you
made a mistake somewhere, gather independent proof that it is correct.

11

RP11: Figures. Try to script all graphs/figures that you create. Yes:
All. Your adviser may ask for a completely different version of a figure, andautomating it prevents lots of manual re-doing. I prefer Matplotlib; it can
output high-quality PDF figures and graphs that can directly be included in
pdflatex.
.

3.3

Research mentality

RM1: The critical reviewer. Often switch roles to a savage reviewer(Mr Hyde) who is looking for any excuse to say: ’I do not believe X; Reject.’Try to identify X yourself and think about which evidence argues for X.
RM2: Your supervisor does not have the answer. We are doing
research. By definition, this research has not been done before. Thus, it is
impossible for your supervisor to give you a list of ToDos: We’ll find them
together.
RM3: Be consistent. Assumptions you make in one part of yourresearch should not suddenly change in another part.
RM4: Question everything. Take a step back, and think about what
you are really doing. Does the story logically make sense? Try to see thethings you take for granted: Is everything justified?
RM5: Simple is strong. Simple is more powerful than complex.
Explain the core of your topic to a smart layperson (your mother?) without
using math/jargon. If you cannot explain it, it is probably too complex.
RM6: Limitations. Identify the limitations of your method. No method
will always be the best. Showing insight where it fails is strong. The goal ofresearch is understanding.
RM7: Write early and often. Writing helps to make thoughts concrete
and it is the interface to your work. Writing always takes longer than you
think, even if you know that it takes longer than you think. Writing isiterative; don’t try to write the perfect text: write a sloppy draft, and iterate.

12

RM8: Not “Eureka” But “That’s funny”.
(by Asimov) is themost exciting phrase in research. It often becomes most interesting when
expectations break.
RM9: Show that the problem exists.
Going directly after improvements is risky: if it doesn’t work, all is gone. Before proposing asolution/improvement: first demonstrate/validate which problem is solved
by it. Demonstrating the problem is valuable in its own right. To demonstrate the problem you are free to choose the setting; a self-constructed fully
controlled (toy) dataset is often ideal. Make sure to validate that a proposedsolution does well on this fully controlled setting.
RM10: Motivate everything.
Novelty is easy: each component canblindly be replaced by another. Always question: why?. Each choice needs
to be motivated with a reason. Is it commonly done? Then give citations.
Is it interesting? Then motivate why. Is it ’obvious’ or speculative? Thenempirically validate it as an hypothesis. (Hitchens’s razor: “What can beasserted without evidence can also be dismissed without evidence”)

13

4

Research meeting questions

Questions during a research meeting often are specific, special case instantiations of more general questions. It is not always clear what goal the specificquestions have, which might confuse, frustrate, or even demotivate. Here, I
aim to give the generalized questions that lie behind specific question types,
with the goal of making it clearer why, and what the goal of a question is.
Additional benefits might include that these questions can be asked even
without a meeting to get unstuck, and might help in preparing and focusing
the meeting. Questions are grouped per stage; where each stage is typicallyrevisited often, see Fig 1.
Research meeting questions
Research question

Related work

Method/approach

RQ1 Why interesting?
RQ2 What storyline?
RQ3 Formalize/simplify
RQ4 What problem?

RW1 Who will use it?
RW2 How different?
RW3 Builds on what?
RW4 Baselines?

MA1 Why this method?
MA2 Explain each step?
MA3 Formalize/simplify
MA4 Alternatives?
MA5 Align with RQ?

Experimental setup

Analyzing

Conclusions

ES1 What Qs?
ES2 How answer Q?
ES3 Baselines?
ES4 Expected outcome?
ES5 Simplify?

AR1 Validate?
AR2 Baseline?
AR3 Understand all?
AR4 When fail?
AR5 Link to Q?

DC1 Exhaustive?
DC2 Expectations?
DC3 Align with RQ?
DC4 Simplify?
DC5 New hypotheses?

4.1

Questions about the Research Question (RQ)

RQ1: Why is the RQ interesting? Why do you care? Who else
cares? Why should others care? Should we change the RQ? It can be usefulto rephrase the RQ to better align with the problem: it can become moregeneral, or more specific. Changing the RQ is normal during the research
process because finding the suitable RQ often takes a large part of theprocess.

14

RQ2: What is the 10-15 bullet point main storyline? Does thestory still make sense? Which point in the storyline are we now discussing?
Is the point still valid? Should we change the storyline? The storyline is the
key motivational driver and changing the storyline is common during theprocess because any new result can invalidated other parts of it.
RQ3: What is the RQ precisely? (Formalize/Simplify) Why
can’t we ask a simpler question? How to formalize it? Which part is themost uncertain? Should we remove that part or should we focus on thatfirst? Formalizing the RQ makes it more precise, possibly revealing hidden
assumptions. Simplifying the RQ can make it more general and/or easier toexplain.
RQ4: Does the problem exist? How often? When (not)? Whatis the simplest example of the problem? How to convincingly demonstratethat the problem exists? Creating a simple, fully controlled, setting of the
problem reveals hidden assumptions. It simplifies and gives focus. Coming
up with a good problem example is difficult, and often takes several iterations.

4.2

Questions about Related Work

RW1: For which work is the RQ interesting? Why can’t thisapproach be used for X? Why can’t Y also make use of this approach?
Should we change the research question to better facilitate X or Y? Don’t
wait for others see links to your work; actively link them yourself.
RW2: How is existing work different/similar? Why can’t method
X already answer the research question? What assumptions are different
from method Y? What other related papers are there? Should we change
the research question to clearly discriminate our setting from X or Y? Or
should align better with X and Y? Motivate where the work fits, how it’sdifferent and how it’s similar.
RW3: Building on what existing work? What is the motivation tobuild on component X? Why don’t we use Y? Can’t we use a simpler ormore common building block? If the building block is not the focus of thework, then it should be as standard as possible.

15

RW4: How does existing work (baselines/competitors) work?
What do baselines/competitors do? Why? How do they solve part X of
the research question? How are we different? Why are we different? There
should be good motivation to do something different.

4.3

Questions about the Method/Approach

MA1: Can you motivate why this method? Each step? Why
not another method? Is each step essential and why? (is there evidence?
ablation?).
MA2: What’s going on? (Visualize output for each step) Whatis it really doing? Please explain this step in detail? Please show only theoutcome of this part, and keep the rest constant. The goal is to validatethat the method is behaving as expected.
MA3: What precisely? (Formalize/Simplify) How to make it
simpler? How to formalize (math?) to describe what exactly is going on?
Simpler is stronger.
MA4: Does method make sense? (Alternatives?) What are otheroptions? Can we motivate why we do not use them? These choices might
not be ’obvious’ and may require an empirical experimental ablation.
MA5: How well is the alignment of the method with the RQ?
Match the method with RQ: Which part of the RQ aligns with this step?
Match the RQ with the method: Where does this part of the RQ comeback in the method? Does it do what we think it does, and if not, should
we then change the RQ to match this? It can be a game changer if resultsinterestingly deviate from expectations.

16

4.4

Questions about the Experimental Setup

ES1: What empirical questions belong to the RQ? What questions
do we wish to answer? Why? What would answering this question give? Do
the questions align with the main RQ? Should we change the RQ to alignbetter?
ES2: What exact question is answered by this experiment?
Why does this experiment answer this question? What other experiments
are possible? Which experiment to do? Each experiment should test an
hypothesis or answer a question.
ES3: What are relevant baselines? Are there very simple (nonlearning? or simple averaging?) baselines to compare to? To which(existing) methods do we compare? Why? Why not more/less? Should we
make the RQ tighter?
ES4: What exact outcome is expected? What outcome is wanted?
Before running the experiment, answer what outcome you would like? What
are the exact numbers that you expect as an outcome? Does doing theanalysis on those numbers give the wanted outcome? If not, should wechange the RQ?
ES5: What is the minimal setting? (Simplify)
How to use asmaller problem? How to use a less complex setting? Why can’t part X be
removed from this setting? Simpler is stronger.

4.5

Questions about Analyzing Results

AR1: How to validate results? How to validate there are no bugs?
How to validate if your method does what you claim it does (semantic
debugging)? Do we have stddevs? Do results consistently align with ourprevious results? Can we do a small test to validate? Can we do an
independent experiment to validate? Bugs are normal, and neural networks
are notoriously difficult to debug; they might seem to work, but there
might still be a mistake. Start with the assumption that there is a mistakesomewhere, and then write code to find it.

17AR2: How to verify correctness of baseline? Do not assume
baselines directly work. How well do we match the reported results in theirpaper? Are these results expected? How to best optimize (hyperparametertuning) the baseline so that the baseline is fairly evaluated?
AR3: Do we understand all results? When looking at the resulttable, can we explain each pattern in the table? Can we also look at someindividual data samples?
AR4: When does it fail? Can we systematically predict when it fails?
Can we look at some individual mistakes? Do these failures make sense?
AR5: Do results answer the question of the experiment? Eachexperiment has a question to answer. What was the question? Do theseresults answer that question?

4.6

Questions about Drawing Conclusions

DC1: What are all conclusions we can draw?
Can we list all
patterns that we see? Can we explain all patterns? Are there patterns that
we have missed?
DC2: How well do results align with previous expectations?
What were the previous expectation? How to explain deviations? Are all
results internally consistent?
DC3: How well do results align with RQ? What results did wewant? How much are these results what we wanted? Should we change the
RQ? Should we redo a different variant?
DC4: Is there a simpler experiment with same conclusions?
Which properties are not essential? Which properties should be moreemphasized? Results/patterns that are not relevant distract from the main
message.
DC5: To which new hypotheses lead these results?
How well dothese align with the RQ? Should we change the RQ to include these new
hypotheses or write them as future work?

18

5

Writing guidelines

Science is also about communicating ideas. This makes writing an essential
part of research. The five most common writing comments that I give backare:• Terms/concepts are not motivated. A new term/concept first has tobe explained and introduced by motivating why the term is important.
• A paragraph has no single topic and no conclusion (ie: no answer tothe ”So What?” question)• Sentences do not follow each other. (Sentence 2 should continue on
topic where sentence 1 ends.)• Figure captions do not explain how to read the figure.
• Figures/Tables have no conclusion in the caption (ie: explicitly write
what should the reader see, and answer ”So What?”)
My writing guidelines are summarized in the table below, and followedby the guidelines themselves.

19

Writing guidelines

5.1

General

StructureForm

WG1 Unburden
WG2 Audience
WG3 Less is more
WG4 No guessing
WG5 Read out loudWG6 More space
WG7 Write as code

WS1 Self-contained
WS2 Consistent
WS3 As discussed
WS4 Paragraphs
WS5 Ref words
WS6 Ref paragraphs
WS7 Latter/Former

WF1 Single topic
WF2 Windows/orphans
WF3 VeryWF4 In order to
WF5 Sort cites
WF6 Brackets
WF7 Synonyms
WF8 Performance

Tables/Figs

Introduction

Related work

WT1 Captions
WT2 Figs are complete
WT3 Tables

WI1 Motivation
WI2 First sentence
WI3 Few research
WI4 Fig 1
WI5 3x contribute

WR1 Subject
WR2 Paragraph
WR3 Layout
WR4 No history lessons

Method/approach

Experiments

Discussion

WM1 Argumentation
WM2 No datasets
WM3 Number eqs
WM4 Eqs are text
WM5 Explain symbols
WM6 Explain eq
WM7 Remove eq
WM8 Define

WE1 Question
WE2 Group
WE3 Analyze
WE4 3 types
WE5 Scale 0-1
WE6 proved
WE7 One more

WD1 Summary
WD2 Limitations
WD3 Conclusions

General writing guidelines

WG1: Unburden the reader. If a reader misinterprets the text: its thewriter’s responsibility. Prof. Freeman: The most dangerous mistake you canmake when writing your paper is assuming that the reviewer will understandthe point of your paper. Avoid that the reader has to do work.
WG2: Audience. Who are you writing for? What is their backgroundand what are they looking for? Help your audience find it. It often helps tokeep a specific person in mind as your writing target.

20

WG3: Less is more. Every word should have a reason to exist. ie:
Remove all unnecessary words. To quote Blaise Pascal: “I would havewritten a shorter letter, but I did not have the time.”. Writing concisely
takes time and effort; it enhances readability.
WG4: No guessing: make it explicit Never expect a reader to doinference. As a writer you need to spell out the thought process for thereader. If the reader has to guess, the guess will often be not what you had inmind. Always explicitly write what the reader is supposed to see/conclude.
WG5: Read out loud. After some time, you will no longer be able toread your own text. Instead, you will read what you meant; not what you
wrote. Tip: read your own writing out loud.
WG6: Important topics take more space.
The more important or
relevant something is to your paper, the more space it takes. If it is not soimportant don’t write too long about it.
WG7: Writing is like coding. Like good code, a paragraph is modular
and self-contained. Do text refactoring just as you would do code refactoring.
Good code is not written in one go, neither is text. Like code, you startwith an initial structure, and restructure several times.

5.2

Structure

WS1: Self-contained.
The reader has not memorized the full text.
Remind the reader of definitions or symbols when defined ’a long time ago’.
WS2: Consistent. Use a defined symbol consistently and uniquely.
WS3: As discussed before, as pointed out earlier, as motivated in
section XXX, as will be described in XXX Avoid using this, it has
no function. The standard sectioning structure of a research paper dictates
where information should be found (main motivation in Intro/Related work;
the technical in the Method; empirical evidence in Experiments, etc.).
Assume your paper will not be read linearly.

21

WS4: Relation between paragraphs. Paragraph topics follow a
logical order. It is helpful to start with a skeleton of topics and keywords. In
addition, creating an inverse outline helps to validate the story. An inverse
outline is created by starting with a text and write down the first andpossibly last sentence of each paragraph: a logical structure should emerge.
WS5: Reference words.
Reference words such as ’this’, ’it’, ’that’,’there’ are often confusing: they require inference/work by the reader whichshould be avoided. Avoid referring: explicitly repeat what ’it’ refers to.
(Here, for example, ’it’ should have been replaced with ’the reference word’).
WS6: Avoid reference words across paragraphs. For example, do
not start a paragraph with ’however’. It is unclear to what you refer to.
WS7:
Avoid ”latter/former”, and ”respectability”. These
reference words require mental ordering and memorization by the reader;
avoid making the reader do this work. Instead, rewrite it without thereference words.

5.3

Form

WF1: A single paragraph, has only one single topic. A paragraphhas an intro sentence to define the topic. Each sentence logically follows theprevious sentence. It has a concluding sentence that concludes the topic: it
answers the “So What?” question.
WF2: Widows and Orphans. Avoid paragraph endings with 1 word
on a new line. Avoid paragraph endings with 1 line on a new page.
WF3: Very.
Do not use “very”. It is supposed to emphasize, yet, it
does the opposite. See also: https://www.proofreadingservices.com/
pages/very.
WF4: In order to. Can almost always be replaced with ’To’.
WF5: Sort citations.
If using numerical citation, make sure not tocite it as [7,2,5], but sort them like [2,5,7] to reduce reader effort.

22

WF6: (brackets). Avoid brackets. If it’s not important remove it; yet,
if it is important, then it should not be in brackets.
WF7: Synonyms.
In non-scientific writing it’s sometimes advised
to make the writing less repetitive by not repeating the exact same terms
and use synonyms. In scientific writing, instead, using a different term forthe same thing will confuse the reader. Choose a single term, and use it
consistently. Clarity trumps eloquence.
WF8: Avoid “performance”. It’s ambiguous; it might mean: speed,
accuracy, memory-use, latency, etc. Choose the precise version you mean.

5.4

Tables/Figs

WT1: Captions in figure/table. Figures and tables should be
self-contained. A reader often starts an article in ’comic book’ mode:
first look at all the pictures. The caption should explain everything tounderstand in the figure/table. Always end with a conclusion to answer the“So what?” question: make explicit what do you want the reader to see here.
WT2: Figures are complete. Label all axis, show the units on theaxis, use a legend with clear differences between entries and add a title toeach (sub)figure. Do not label sub-figures (a), (b) and explain what (a) and(b) are in the caption: instead label each sub-figure with a title. Do not use
too thin lines or too small of a font.
WT3: Tables.
For formatting tables, read section 2 of this document:”booktabs tables” and update your tables accordingly.

5.5

Introduction writing

WI1: Motivation and scope. The intro starts with a “just broad
enough” motivation; not too broad, and not too specific, then quickly
narrows the scope smaller, and smaller, culminating to exactly your topic.

23

WI2: No generic first sentence. The fist sentence of the introduction
should focus/engage your audience. Don’t use a sentence that can be added
to any other paper in your field. Test if it would still make sense if you leaveout the sentence, and, if it does, then leave the sentence out.
WI3: 3 contributions. Rule of thumb: your paper has 3 contributions.
A contribution is something that a peer researcher will find interesting.
Do not expect the reader to do inference work, so end the introduction by
explicitly stating your contributions.
WI4: Figure 1.
Make a visual abstract of the paper in Figure 1. Bestif this is the main idea, but it can also be a pipeline figure.
WI5: Few research.
Just because a topic has seen ’little research’is by itself not a good motivation (For example: my right thumb has notbeen researched at all, but its still uninteresting.). Don’t motivate by what
others have not done –put that in related work– instead: motivate what isinherently interesting about your research, ie: what problem does a possible’user’ have, and what can be gained by reading this paper? What would apeer-researcher find interesting to learn?

5.6

Related Work writing

WR1: The subject is the method, not the paper.
Do not write:
The important work of [a] does X which is followed by the work of [b] that
does Y. It has papers as the subject. Instead, make the method the subject
and add citations to the method: X [a] is important, and extended by Y [b].
WR2: Paragraph topic.
One paragraph in the related work section
is grouped around a single topic. Related papers often have multiple topics.
It is up to you to group related work as best for you. Rule of thumb: Eachparagraph has 3-10 citations.
WR3: Paragraph layout.
The first sentence defines the scope. Then,the following sentences, you group papers based on what they do. The final,
concluding, sentence is how are these methods related to your method. You
have two options: option 1. All so great, we make use of it. Option 2. All isgreat, but we do something different because . . .

24

WR4: No history lessons.
Related work means related to yourresearch question. Avoid lengthy ’general history lessons’ about methods/concepts that are more general than your research question. It is OK
to give some history of work related to your research question. The goal of
the related work section is about motivating why you choose the works thatyou build on, and why you choose the works that you contrast with.

5.7

Method writing

WM1: No general argumentation. The method should only motivateand explain the technical method. All argumentation for the main idea
should be in the Introduction or in the Related Work sections.
WM2: No datasets. Datasets and their description belong in theexperiments. Only a toy problem is allowed in this section if it helps to
explain the method. The method section only explains the technical part ofthe method. (Exception: when you are writing a dataset paper)
WM3: Number all equations.
Maybe you do not refer to them,but others (reviewers, readers) may want to. Only if the equation is notessential, it is OK to have it in the flow of the text without a number.
WM4: Treat equations as normal text.
If an equation ends thesentence, the equations ends with a period. If the sentence continues, use acomma after the equation.
WM5: Explain all symbols.
Directly before, or directly after
introducing an equation: all symbols should be explained. A formula
should be self-contained: the reader should be able to understand it without
searching for symbol definitions elsewhere.
WM6: Explain equation in words.
Directly before giving anequation, first explain why and what you aim to achieve in English. This
makes the following equation easier to follow.

25

WM7: Should be understandable without equations. The methodshould also be understandable without reading the equations. Motivate anddescribe in English what is happening, the equations make the words exact.
Test if you can still get the main point of the paper when all equations are
removed.
WM8: Only define the relevant.
Do not define everything
you can think of, only define things you will actually use. Best to firstwrite the results, and later define only what is needed to obtain these results.

5.8

Experiments writing

WE1: Answer a question.
Every experiment starts with a question.
Explicitly write this question. The experiment should answer that question.
WE2: Group an experiment together.
Make use of sub-sections
or bold-words, to help the reader understand the structure of the section.
Each experiment is grouped as a module. Give each experimental question
a number: ”Experiment 1: How X applies to Y”.
WE3: Analyze results modular.
Every experiment has its own
tables/figures for the results. Do not mix experiments by grouping all results
in a single huge table. Group results together that are compared together.
It may mean you have to repeat values in multiple tables/figs.
WE4: Experiment types. Broadly speaking there are three types ofexperiments. 1: Validate: does it do what you claim it does, (fully controlled
setting?). 2: Investigate: what unique properties does your method have.
3: Compare: how does it compare to others. Present them in that order.
WE5: Scale scores between 0 and 1. Avoid useless zeros, scales 0-1
scores to 0-100. E.g.: ’0.07’ becomes ’7’.
WE6: Proved. Experiments do not prove. A proof is derived in math,experiments demonstrate empirically for the setting at hand.
WE7: One more. When you think you are done, see if you can add one
more experiment to show relevance for a different domain or application.

26

5.9

Discussion writing

WD1: Summary.
context.

Small summary of what you did to highlight the

WD2: Limitations.
No method will always be the best. Showing
insight where it fails is strong. The goal of research is understanding.
WD3: Conclusions.
“Great paper; but So What?”. Answer thisquestion to draw conclusions. Don’t make too broad conclusions; keep itmodest and factual, and at the same time don’t shy away in mentioning
what is interesting and why.

27

6Giving a Talk

Make a deliberate choice for the medium for your talk (whiteboard/slides/...).
This document assumes that slides are used. Also see the writing guidelines;
several are also applicable to giving a talk.
Motivation

Content

Form

Analysis

TM1 Goal
TM2 Audience
TM3 Refresh
TM4 Unburden

TC1 1 slide 1 topic
TC2 Less is more
TC3 Self-contained
TC4 Define terms
TC5 No guessing
TC6 Multi-modal

TF1 Too much
TF2 No TOC
TF3 Layout
TF4 No sentences
TF5 Animate
TF6 Complete figs
TF7 Number slides

TA1 Exps answer Q
TA2 Limitations
TA3 Peer review

6.1

Motivation of your presentation

TM1: What is the goal? What you want to get out of it. There is areason for giving your presentation: What is it? (and, no, it is not: ’it is myturn’ or ’they told me to’). It may help to share this reason with the audience.
TM2: Audience.
Whom are you presenting for? What do you want
the audience to take away? What is their background and what are theylooking for? Help your audience find it. Avoid Jargon.
TM3: Refresh. Always start with 1 or 2 slides (re-)introduction. Do notassume your audience will remember anything from your last time; there
may also be new viewers present. If it is important: briefly repeat it.
TM4: Unburden the audience. If the audience misinterprets themessage: its the responsibility of the presenter to reduce the understanding
effort. Audience understanding can be validated by asking them.

6.2

Content of the presentation

TC1: A single slide has a single topic. A slide has a title to scope thetopic. It has a concluding phrase that makes the main point of the topic.

28

TC2: Less is more. Every word/figure/image should have an explicit
reason to exist. Do this test: Can I safely remove it yes or no? Do not put
information on the slide that you don’t want to answer questions about (eg:parts of a figure you took from another paper). Presenting concisely andprecisely takes time and effort; it enhances understanding.
TC3: Self-contained.
The audience has not memorized the fullpresentation. Remind the audience of definitions or symbols when defined’a long time ago’.
TC4: Define terms.
Define all symbols/terms.
symbol/term consistently and uniquely.

Use a defined

TC5: No guessing Never expect the audience to do inference. If theviewer has to guess, the guess will often be not what you had in mind.
Always explicitly write what the viewer is supposed to see/conclude. Ie: put
the answer to the ”So What” question on the slide.
TC6: Multi-modal There are various people in the audience whose
preferences range from visually, formulas, auditory. Be sure to present a mix.

6.3

Syntax, layout and form

TF1: Do not present too much. A rough guideline: at least 1 minuteper slide.
TF2: Do not use a table of contents. Avoid the default toc of“• Intro, • Method, • Exps, • Conclusion”; this is expected, so this toc adds
nothing. Another form (e.g.: visual abstract) can be useful.
TF3: Good layout eases the viewer’s effort. Use the full screen. Be
consistent. Not too much info in a slide.
TF4: Do not write long sentences. Use bullet points with one phrase
per point. One phrase fits on a single line. Correct grammar is secondary,e.g., there is no need for complete sentences with a subject, a verb, etc.

29

TF5: Only use animations sparingly and IFF they add value. For
example, to emphasize, or to prevent overload by iteratively making more
content appear.
TF6: Figures are complete. Label all axis, show the units on the axis,
use a legend with clear differences between entries and add a title to each(sub)figure so that the reader can directly see what is shown. Do not use
too thin lines or too small of a font: It has to be seen from the back of theroom. Add the conclusion you would like the viewer to draw.
TF7: Number your slides so that viewers can refer to them.

6.4

Presenting analysis

TA1: Experiments answer a question.
If you present experiments,
note that every experiment starts with a question. Write the question on
the slide (maybe in the title?). The experiment should answer that question.
Write the answer on the slide.
TA2: Limitations.
What are the limitations of your method. No
method will always be the best. Showing insight where it fails is strong.
The goal of research is understanding.
TA3: Peer review.
Find a peer to review each others presentations.
Check if their presentation follows these guidelines. Keep in mind that if
an honest viewer did not understand it, then the presentation should be
improved (not the viewer).

30

7Poster guidelines

Poster presentations are common for presenting research ideas. I’ve foundthis blog useful: How to design an award winning poster. I prefer to do
my posters in inkscape. Several of the writing guidelines and guidelines forgiving a talk also apply here.
Motivation

Content

Form

Analysis

PM1 Excite
PM2 Audience
PM3 Refresh
PM4 Unburden

PC1 1 block 1 topic
PC2 Less is more
PC3 Self-contained
PC4 Define terms
PC5 No guessing

PF1 Too much
PF2 Reading order
PF3 Layout
PF4 No sentences
PF5 Draw attention
PF6 Complete figs
PF7 Find examples

PA1 Exps answer Q
PA2 Limitations
PA3 Peer review

7.1

Motivation of your poster

PM1: Excite the viewer The goal of a poster is to advertise yourresearch so people will want to read your report/paper. Excite us!
PM2: Audience.
Whom are you presenting for? What do you want
the audience to take away? What is their background and what are theylooking for? Help your audience find it. Avoid Jargon. What do you wantto get out of it from them?
PM3: Refresh. Do not assume your audience will have remembered
anything from any other source; there may also be new viewers present. If a
topic is important: briefly repeat it.
PM4: Unburden the audience. If the audience misinterprets themessage: its the responsibility of the presenter to reduce the effort ofunderstanding. Audience understanding can be validated by asking them.

31

7.2

Content of the presentation

PC1: A single block has a single topic. A modular block in yourposter has a title to scope the topic. It has a concluding phrase that makes
the main point of the topic.
PC2: Less is more. Every word/figure/image should have an explicit
reason to exist. Do this test: Can I safely remove it yes or no? Presenting
the core essence takes time and effort; it enhances understanding.
PC3: Self-contained.
The main point of the poster has to beunderstandable without a presenter. While you are busy explaining yourposter, another viewer who just walks in should be able to understand the
key idea without your help.
PC4: Define terms.
Do not assume the audience will know specificsymbols/terms/abbreviations. Use a defined symbol/term consistently anduniquely. All terms in an equation should be explained.
PC5: No guessing Never expect the audience to do inference. If theviewer has to guess, the guess will often be not what you had in mind.
Always explicitly write what the viewer is supposed to see/conclude (answer
the “So What” question).

7.3

Syntax, layout and form

PF1: Do not present too much. Your goal is to advertise and excite.
Nitty gritty details should be in the report not in the poster. Show justenough so a reader can follow, no more. Do not overwhelm, do not even try
to be complete with all details: it will scare people away.
PF2: Use numbers to show reading order. Make explicit how you
wish your poster to be read: Numbers the reading order.
PF3: Good layout eases the viewer’s effort. Be consistent. Keep somewhite-space, don’t scare people away with an avalanche of detail.

32

PF4: Do not write long sentences. Use bullet points with one phrase
per point. One phrase fits on a single line. Correct grammar is secondary,e.g., there is no need for complete sentences with a subject, a verb, etc.
PF5: Draw attention. Make your poster visually stand out from allothers. The goal is to advertise.
PF6: Figures are complete and have a conclusion. Label all axis,
show the units on the axis, use a legend with clear differences between
entries and add a title to each (sub)figure so that the reader can directly see
what is shown. Do not use too thin lines or too small of a font. Always add
the conclusion you would like the viewer to draw.
PF7: Example poster. Find some scientific posters on the internet andapply these guidelines. Make a list of things you do/do not like in a poster.

7.4

Presenting analysis

PA1: Experiments answer a question.
If you present experiments,
note that every experiment starts with a question. Write the question on
the poster. The experiment should answer that question. Write the answer
on the poster.
PA2: Limitations.
If applicable: What are the limitations of yourmethod. No method will always be the best. Showing insight where it fails
is strong. The goal of research is understanding.
PA3: Peer review. Find a peer to review each others posters. Check if
their poster follows these guidelines. Keep in mind that if an honest viewer
did not understand it, it is the mistake of the presenter.

33

8

Reviewing and rebuttal guidelines

Why review. If you are writing peer-reviewed papers, then you are asking
others to devote time and effort to your work. Thus, it’s fair to return thefavor and review the work of others. Reviewing is part of your academic
community. It offers other advantages: improving the scientific field, learning
something new, practicing your critical thinking, and helping others.
What to review. I’ve found the “Troubling Trends in Machine Learning Scholarship” [6] paper quite helpful, where I regularly give back a review where I state that a paper is Confusing explanation with speculation
and/or has a Failure to identify the sources of empirical gains. In addition,
I use Hitchens’s razor: “What can be asserted without evidence can also be
dismissed without evidence. And, reviewing involves applying the researchguidelines in this document, albeit not during the process, but at a finishedpaper. Specifically, see if you can find the storyline as in Section 2, look
for the answers that can be asked during a research meeting in Section 4,
and the writing guidelines in Section 5. I’ve labeled each guideline with aunique identifier, which could help in motivating a review by referring to thisdocument and it’s identifiers.
Goal of reviewing. The main goal of reviewing is to improve the workof others by giving feedback while preventing the publication of flaws. Thismight include flaws in the following. Hypothesis: Are the hypotheses sound?
Literature: Are the relation to relevant other work present, correct and properly motivated why and how the work is related?. Method: Is the methodaligned with the hypotheses? Technical: Are the equations correct? Does it
do what is claimed? Are there no unexplained surprises? Is it reproducible,
ie: code? Experimental setup: are the hypotheses evaluated? Is the motivation evaluated? Evaluation: is it an unbiased, fair, comparison to others?
Clarity: is it understandable? Figures/tables readable? it’s OK if there areminor spelling/grammar mistakes, as long the paper can reasonably be wellunderstood without too much puzzling. As a reviewer, it typically cannot be
expected to rerun experiments, a review is inherently based on trust in theauthor’s integrity. A perfect paper does not exist, all papers are limited insome sense. Thus, be critical, but appreciate the positives.

34

What is a good paper. It’s a solid brick that others can build on: something is learned. It’s well written with an intuitive motivation, for example
in Fig 1. It has clearly specified hypotheses, research questions, and contributions. The method aligns well with hypotheses. The Hypotheses, researchquestions and contributions are backed up by empirical evidence. Comparison experiments vary only 1 variable. It has experiments on several datasets
to illustrate generalization. Bold numbers are never a goal in itself, they
are ’only’ important to show relevance/usefulness. It’s reproduceable, it has
clear algorithms, or better: code.
Addressing novelty. It’s easy to do something novel: merely add a layer,
and it’s ’novel’. Novelty is not a goal in itself: a paper about my left thumbis extremely novel, but that does not make it, nor the paper, interesting. It is
up to the paper under review to explain in the introduction and related work
sections how it relates to others, and what contributions it has compared tothe other work. Just because the outcome is ”obvious”, or ”trivial” is notgood grounds for rejection: the paper has now confirmed this outcome; andthis confirmation is a contribution in itself. A possible ground for rejection
could be if there is other near-identical work but not placed in relation to
the paper under review and/or if it is not experimentally compared against.
Review quality. A bad review: makes claims without giving details, citations, or motivation. Is only a few lines. Only checks the bold numbers.
A good review gives constructive author feedback, so in addition to What,it also suggests How to change a paper. Is well motivated, with detailed
justification (citations / line numbers). Is well-written and self-contained:
the review is readable without the paper. It makes the decision for the ACeasier.
My review structure Review formats vary slightly for each conference.
I always use this layout in a .txt file, which can be poured in any format.
When I am reading the paper for the first time, I directly write comments
per line. Once these detailed comments per line number are there, then theother points follow from them. Review structure:• Summary. Unbiased, the authors should agree with it, introduce terms
that you will build on later so that the review can be self-contained.

35• List of positive points: just 1 line per point• List of negative points: just 1 line per point• A conclusion paragraph of score motivation and main suggestions forwhat the paper could address in the rebuttal. This paragraph builds
on the summary and the list of positive/negatives.
• Detailed comments per line number with detailed justification.

8.1

Rebuttal guidelines

Why write a rebuttal. Several conferences, and journals, allow for a
rebuttal: a factual response to the reviewers. The main goal of a rebuttal is
to correct mistakes, and convince the reviewers your work is interesting. Even
if there is only 1 positive reviewer it can help to write a rebuttal: there often
is a discussion phase, where your ”champion” can then defend the paper.
Not novel Typically, when reviewers write that there is insufficent novelty–without citing a missing paper– then what they really mean is that, unfortunately, they did not find the paper interesting. Yes; finding somethinginteresting is subjective; and this is probably why they write ’not novel’ instead of ’I didn’t find it interesting’. I suspect that reviewers are afraid to behonest because it is not possible to give objective reasons for why something
is not interesting (to you). This is unfortunate, because as an author thiswould be valuable feedback to have. If reviewers find it not novel, then try to
ask yourself why they didn’t find it interesting. Try to also ask a (somewhat
senior?) not co-author colleague, who is not afraid to tell you the truth toyour face.
Science is done by humans Many reviewers: do their work too fast;
have a rejection mentality; do not read the paper well; write a too shortand unmotivated review; or base the review on the author’s name if a preprint or blog is available. Receiving a ’bad review’ can be quite frustrating;especially when you spent all that effort on your paper. The only advice I
can give is try to learn something from the review anyway. If the reviewer
did not understand the paper: what can be improved? How can the paper bemade easier to parse? How to improve ”something between the lines” thatthey did not like?
36

Because science is done by humans, its also important to address thereviewers as human beings:• Always thank the reviewers (Don’t “over-thank”).
• Assume they will not change their mind more than 1 point (It might
happen, but is psychologically difficult)• If technically possible: Do what they ask, even if it doesn’t make sense(to you). The most convincing response is to just show it.
• If you fight/shout/insult: they will fight back in the discussion; andthey will have the last word.
• Write for reviewers and area-chair/editor; having the reviewers on yourside is much easier to get accepted.
• Make it easy for the reviewer to find their answer (do not ’hide’ theanswer somewhere in a lot of text). Thus, try to copy the comment of
the reviewer verbatim• Write the rebuttal self-contained: ideally, they should not need to go
back to the paper, nor to any of the reviews.
• Do not take reviews personal (you are not your work)• Reply positive, non-defensive and to the point• Be polite and professional, but self-assured and firm
• Long and too dense rebuttals will scare reviewers away. Leave sufficientwhite space.
My approach to writing a rebuttal is as follows:
1. Copy-paste all concrete positive and negative points in a doc
2. Answer each negative point
3. Perform all requested experiments (to good approximation)
4. Group similar (positives and negative) points

37

5. Start by summarizing grouped (verbatim) positive comments
6. Answer grouped (verbatim) negative comments
7. Rephrase negative answers and compactly rewrite
8. Decide which answers to drop strategy (eg: Convince one reviewer, but
keep others).
9. Ask someone else to read rebuttal and ask how they feel38

9Appendix: Separately printable tables.

Each guideline is uniquely labeled, which can be used to give feedback, during
a meeting, or to use as a reminder. Below I copied all tables on a separate
page for convenient printing and adding your own guidelines.

39

Organization

Process

Mentality

RO1 Full responsibility
RO2 No dependencies
RO3 Meet advisor
RO4 Focus advisor
RO5 Take critique
RO6 Constructive disagree
RO7 Analyze results
RO8 Suggest solutions
RO9 Give feedback
RO10 Safety

RP1 One main Q
RP2 Min. 3rd party
RP3 Validate baselines
RP4 First break it
RP5 Depth first
RP6 Exps answer Q
RP7 Proof of concept
RP8 Exps max 1 night
RP9 Change 1 var
RP10 Debug science
RP11 Figures

RM1 Be critical
RM2 Find todos together
RM3 Consistency
RM4 Question everything
RM5 Simple is strong
RM6 Embrace limitations
RM7 Write early and often
RM8 Not eureka
RM9 Show the problem
RM10 Motivate everything40

Research meeting questions
Research question

Related work

Method/approach

RQ1 Why interesting?
RQ2 What storyline?
RQ3 Formalize/simplify
RQ4 What problem?

RW1 Who will use it?
RW2 How different?
RW3 Builds on what?
RW4 Baselines?

MA1 Why this method?
MA2 Explain each step?
MA3 Formalize/simplify
MA4 Alternatives?
MA5 Align with RQ?

Experimental setup

Analyzing

Conclusions

ES1 What Qs?
ES2 How answer Q?
ES3 Baselines?
ES4 Expected outcome?
ES5 Simplify?

AR1 Validate?
AR2 Baseline?
AR3 Understand all?
AR4 When fail?
AR5 Link to Q?

DC1 Exhaustive?
DC2 Expectations?
DC3 Align with RQ?
DC4 Simplify?
DC5 New hypotheses?

41

Writing guidelines
General

StructureForm

WG1 Unburden
WG2 Audience
WG3 Less is more
WG4 No guessing
WG5 Read out loudWG6 More space
WG7 Write as code

WS1 Self-contained
WS2 Consistent
WS3 As discussed
WS4 Paragraphs
WS5 Ref words
WS6 Ref paragraphs
WS7 Latter/Former

WF1 Single topic
WF2 Windows/orphans
WF3 VeryWF4 In order to
WF5 Sort cites
WF6 Brackets
WF7 Synonyms
WF8 Performance

Tables/Figs

Introduction

Related work

WT1 Captions
WT2 Figs are complete
WT3 Tables

WI1 Motivation
WI2 First sentence
WI3 Few research
WI4 Fig 1
WI5 3x contribute

WR1 Subject
WR2 Paragraph
WR3 Layout
WR4 No history lessons

Method/approach

Experiments

Discussion

WM1 Argumentation
WM2 No datasets
WM3 Number eqs
WM4 Eqs are text
WM5 Explain symbols
WM6 Explain eq
WM7 Remove eq
WM8 Define

WE1 Question
WE2 Group
WE3 Analyze
WE4 3 types
WE5 Scale 0-1
WE6 proved
WE7 One more

WD1 Summary
WD2 Limitations
WD3 Conclusions

42

Presentation guidelines
Motivation

Content

Form

Analysis

TM1 Goal
TM2 Audience
TM3 Refresh
TM4 Unburden

TC1 1 slide 1 topic
TC2 Less is more
TC3 Self-contained
TC4 Define terms
TC5 No guessing
TC6 Multi-modal

TF1 Too much
TF2 No TOC
TF3 Layout
TF4 No sentences
TF5 Animate
TF6 Complete figs
TF7 Number slides

TA1 Exps answer Q
TA2 Limitations
TA3 Peer review

43

Poster guidelines
Motivation

Content

Form

Analysis

TM1 Goal
TM2 Audience
TM3 Refresh
TM4 Unburden

TC1 1 slide 1 topic
TC2 Less is more
TC3 Self-contained
TC4 Define terms
TC5 No guessing
TC6 Multi-modal

TF1 Too much
TF2 No TOC
TF3 Layout
TF4 No sentences
TF5 Animate
TF6 Complete figs
TF7 Number slides

TA1 Exps answer Q
TA2 Limitations
TA3 Peer review

44References[1] Serge Demeyer. Tutorial: Research methods in computer science. https://win.uantwerpen.be/~sdemey/Tutorial_ResearchMethods/, 2010.
[2] Tom Dietterich. Research methods in machine learning. http://web.
engr.oregonstate.edu/~tgd/talks/new-in-ml-2019.pdf, 2019.
[3] Paul Feyerabend. Against Method: Outline of an Anarchistic Theory ofKnowledge. New Left Books, 1975.
[4] Sam Greydanus.
Scaling down deep learning.
arXiv:2011.14439, 2020.

arXiv preprint[5] Thomas S Kuhn. The Structure of Scientific Revolutions. The Universityof Chicago Press, 1962.
[6] Zachary C. Lipton and Jacob Steinhardt. Troubling trends in machinelearning scholarship. arXiv, 2018.
[7] Julian Togelius and Georgios N. Yannakakis. Choose your weapon: Survival strategies for depressed ai academics. arXiv, 2024.

45