Penetration Test Report (CPTC): letter to competitors

To All CPTC Competitors:
As organizers of CPTC, our core goal is to foster education and help the next generation of
offensive security practitioners gain practical experience that they can use in their future
careers. To support this mission, we’ve been increasing the amount of feedback and
information that we share with competitors in order to improve every team which participates in
our event. Sadly, our time and resources are limited and not every team has the opportunity to
make it to the final round of competition and experience all that CPTC has to offer, limiting that
goal.
Broadly, reporting is one of, if not the single most important part of professional penetration
testing. Our clients and customers are paying for our expertise, experience, tools, and
techniques; however, the only lasting results we leave with them are their impression of our
team and the documentation of issues: typically, our report.
Frustratingly this means that many different levels and parts of the organization are the targets
of our reports; often with competing requirements. For example, a high level executive may
look at deeply technical information incorrectly and get frustrated, worried, or have numerous
questions that are outside their scope of expertise while a systems engineer often needs
extensive detail to resolve an issue correctly. Reports need to be written for the Board of
Directors, CIO, upper management, IT management, engineers, developers, risk managers, IT
analysts, as just a few examples. Because of this you will often find multiple sections of a report
attempting to aim for the needs of different groups.
Sadly, this also means that different groups will get confused by sections not meant for them.
Different individuals will also give varying feedback based on their experience. This is also true
when looking at CPTC scoring: one volunteer may grade something as a positive while others
find it a negative. This is one of the most real-world aspects of the competition and is one of the
main reasons we look for volunteers to score reports and presentations that are diverse in
technical skill, organizational level, background, experience, and focus.
In our experience as professional penetration testers outside of CPTC, we also find this is an
area almost all of our newest, least experienced team members struggle. Competing
requirements and changes from team to team and client to client are an early career frustration
for many, but over time they learn the art of report drafting that begins to balance these needs.
In the case of CPTC, there are some specific feedback and examples that we wanted to share
based on our experience grading CPTC reports:
● Do include technical evidence that you were able to perform an issue. While this may be
​
different in a professional setting, where you can have multiple deliverables, at CPTC
you have one, and we need to be able to reperform what you’ve done or see that you’ve
done it clearly.
● Don’t include unredacted sensitive customer, employee, or intellectual property
​
information in reports. Strive to aim higher than your client, who may have been terrible
at storing information securely:
● Do include references to findings or relevant information to help your audience locate
​
information and solve issues, such as technical links, example commands or code to fix
the issue, etc.
● Don’t use hyperlinks where without the full link, as reports are often printed or converted
​
to PDF.
● Do use professional, third person, language and provide an explanation for everything
​
you include, a screenshot on it’s own doesn’t communicate as much as one with context.
● Don’t personally blame the client for any issues identified. Avoid second-person
wording, such as phrases beginning with the word “you”.
● Do include an executive summary, and make sure it’s appropriate for the intended
​
audience and level.
● Don’t include excessive technical detail in an executive summary. This audience may
not know what an IP address is, let alone care about it. This will often confuse the reader
and cause more questions and concern than necessary.
● Do recognize that your client will have questions about your findings, and you will be
asked to reperform some activities within client environments to validate their
remediation. Clients that partner with you throughout the testing process are the clients
you will want to work with year after year and are the ones that will often want to work
with you.
● Don’t just show the client how they messed up, show what they did well to give them
some credit. A positive start can make a massive overall difference in how your message
is received.
● Do always seek to learn and improve. The information security field is a relatively small,
tight-knit community - and your reputation for quality work and integrity are incredibly
valuable.
● Don’t forget to proofread and spell check your report. These errors undermine your
​
credibility and make your overall message less professional.
● Do provide reports in formats where they cannot be easily modified. Customers may
want certain findings omitted from reports, but you should only document the resolution
of these findings once they’re remediated if you have validated the remediation and are
still within the timeframe of your engagement. In general, findings should never be
removed from a report as without the engagement they would have not been identified
and resolved.
● Don’t share your report with a client in an insecure way. Malicious actors would love to
get their hands on one, documentation on all the weaknesses will save them effort..
Today, we’re excited to announce another step to help competitors and the information security
community as a whole: releasing all reports submitted for the International Finals event of the
2019 season of CPTC. While these reports have been anonymized to remove any remaining
team names and numbers, they represent the final deliverables from each team during this
event. None are perfect, but are great examples for you to review to find the parts that you like
the best. Every report, from CPTC to professional, is different. We look forward to seeing the
new approaches you all have this year!
Thank you all for everything you do,
The CPTC Volunteers and Directors