# Stress Testing the DMARC Reporting System: Compliance with Standards and Ways of Improvement

Olivier Hureau, Andrzej Duda, Maciej Korczyński

► To cite this version:

Olivier Hureau, Andrzej Duda, Maciej Korczyński. Stress Testing the DMARC Reporting System: Compliance with Standards and Ways of Improvement. CoNEXT '24: The 20th International Conference on emerging Networking EXperiments and Technologies, Dec 2024, Los Angeles, CA, United States. pp.1-9, 10.1145/3680121.3697809. hal-04832277

HAL Id: hal-04832277

https://hal.science/hal-04832277v1

Submitted on 11 Dec 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Stress Testing the DMARC Reporting System: Compliance with Standards and Ways of Improvement

Olivier Hureau

Univ. Grenoble Alpes, CNRS,
Grenoble INP, LIG
Grenoble, France

hureauo@univ-grenoble-alpes.fr

Andrzej Duda

Univ. Grenoble Alpes, CNRS,
Grenoble INP, LIG
Grenoble, France

dudaa@univ-grenoble-alpes.fr

Maciej Korczyński

Univ. Grenoble Alpes, CNRS,
Grenoble INP, LIG
Grenoble, France

korczynm@univ-grenoble-alpes.fr

## Abstract

The DMARC reporting system enables domain name owners to receive failure and aggregate reports from email receivers to get insight into email authentication failures or unauthorized domain usage and take measures against phishing and spoofing attacks. In this paper, we test DMARC reporting of main Email Service Providers (ESPs) and show that most of them do not comply with the RFC specification, especially with respect to the generation of failure reports and external destination verification. We also analyze more than 40,000 DMARC aggregate reports and show that 94.93% of them do not follow the correct report syntax defined in RFC 7489. Finally, we provide recommendations for improving the DMARC reporting strategy.

## CCS Concepts

* Networks → Protocol testing and verification.

## Keywords

Email Anti-Spoofing mechanisms; DMARC; SPF; DKIM

## 1 Introduction

The DomainKeys Identified Mail (DKIM) and the Sender Policy Framework (SPF) are two main email authentication mechanisms. In addition, Domain-based Message Authentication Reporting and Conformance (DMARC) provides a means for specifying how mail receivers should treat the emails that fail authentication checks by DKIM and SPF: domain owners configure the DMARC policies for their domain names to define the actions for the email receiver to take if the results of the DKIM and SPF verification do not succeed. If the DMARC policies are too strict or incorrectly configured, there is a danger of losing unauthenticated but legitimate emails. Actually, the proper operation of email systems requires continuous refinement to eliminate spoofed emails while keeping the probability of losing legitimate emails low. This is the role of the DMARC reporting system that provides the required feedback to domain owners to improve the configuration of their email systems.

The DMARC reporting system is crucial for the operation of email systems: if email receivers do not provide suitable feedback, domain owners cannot properly identify and fix problems in their configurations. Moreover, if the email receivers that generate reports do not comply with the specification of the report format, domain owners face problems with their automated analysis.

In this paper, we present the results of stress testing Email Service Providers (ESPs) from the point of view of DMARC reporting. In the first experiment, we use a comprehensive methodology to

evaluate how 25 most important ESPs use DMARC reporting. The measurements show that most of the tested ESPs do not comply with the DMARC reporting system specification, especially with respect to the generation of failure reports and external destination verification. We also test the extent to which ESPs follow the DMARC reporting recommendations defined by RFC 7489: we analyze over 40,000 DMARC aggregate reports collected at a large European university and observe that if sent, most of the DMARC reports are not correctly formatted—94.93% of them do not follow the correct report syntax defined in RFC 7489. Finally, we recommend redefining the reporting strategy outlined in the current RFC specification. We advocate standardization of the DMARC aggregate report format for interoperability across all versions.

## 2 Background

This section briefly introduces DKIM, SPF, and DMARC.

### 2.1 DKIM

The goal of DKIM [7] is to ensure the integrity of emails with digital signatures. The header of an email includes the signature (the DKIM-Signature attribute), the Signing Domain Identifier (SDID, the d tag), and the key selector (the s tag). The SDID is the domain name that signs the email, thereby claiming the responsibility for its content. It is not mandatory for the SDID to match the author domain (i.e., the domain name of the email address in the 'FROM' header field of the email). The email receiver obtains the public key by querying the TXT record for the `<selector>.<domainkey.<SDID>.example.com` domain. The DKIM verification results in **success**, **permfail**, or **tempfail**.

### 2.2 SPF

SPF [22] authenticates the sender by checking the name specified in the 'MAIL FROM' address ('MAIL FROM' identity) and the domain in the 'HELO/EHLO FROM' command ('HELO/EHLO FROM' identity) during Simple Mail Transfer Protocol (SMTP) transactions. The email receiver executes the `check_host()` function to retrieve the TXT record of the fully qualified domain name (FQDN) in the 'MAIL FROM' and 'HELO/EHLO FROM' identities, and checks if they are authorized to send emails on behalf of the domain. The output of the function is **neutral**, **pass**, **fail**, **softfail**, **tem**, or **permerror**. DMARC only considers the result of `check_host()` on the 'MAIL FROM' identity.

### 2.3 DMARC

DMARC [25] aligns the SPF and DKIM verification results and the policies defined by the owner of a domain name in the
<figure><img src="image_2.png" /><figcaption>Figure 1: Example of the DMARC record stored at _dmarc.example.com for the example.com domain.</figcaption></figure>

DMARC record—a TXT DNS record of the _dmarc subdomain: for the example.com domain, its DMARC record is located at _dmarc.example.com. The record starts with ‘v=DMARC1’ and includes a list of tag-value pairs separated by semicolons as illustrated in Figure 1. Moreover, Figure 3 in Appendix A illustrates the email ecosystem with an example of the entities involved in the DKIM, SPF, and DMARC processing.

2.3.1 DMARC Check Mechanism and Alignment. DKIM and SPF cannot completely prevent email spoofing—DKIM ensures the email integrity, and SPF checks if the email comes from an authorized host. However, neither scheme directly connects the signing domain (the case of DKIM) or the sending server (the case of SPF) to the domain appearing in an email as the author (i.e., the author domain). The role of DMARC is to verify if DKIM or SPF verified domains are aligned with the author domain. If the domain owner sets the strict mode for DKIM or SPF (**adkim** or **aspf** tag), the domains in the DKIM or SPF check output must match the author domain. If the **aspf** or **adkim** tags are set to relaxed (default options), the domain in the DKIM or SPF output must be in the same organizational domain name. For instance, with the ** mail.pmetal.com** author domain and the **mail.pmetal.com** ‘MAIL FROM’ domain in SPF, SPF is considered aligned when **aspf** is in the relaxed mode but not aligned when in the strict mode. If both DKIM and SPF are not aligned, the DMARC check mechanism **fails**. The DMARC handling policies define the actions to undertake by the email receiver when the DMARC check mechanism **fails**.

2.3.2 Handling Policies. The p and sp tags specify the handling policies of a domain: their values of none, quarantine, or reject determine how emails failing the DMARC checks should be treated. Although the email receiver ultimately decides how to handle in-coming emails, the domain owner specifies the preferred behavior.

If the p value is set to none, the domain owner does not mandate any specific action by the email receiver. In the case of quarantine, the email receiver should treat the email as suspicious, and when the policy is reject, the email receiver should reject the message. The p value defines the policies for the domain and subdomain, unless an sp tag is present, thus defining the policy for subdomains.

2.3.3 DMARC Reporting System. DMARC specifies a reporting system based on the rua and ruf tags that contain email addresses to which the email receivers should send the reports on the domain name email activity. The email receiver should send aggregate reports to email addresses present in the rua tag on a daily basis (or with the frequency specified in the ri tag). Aggregate reports are composed of different parts: i) the DMARC policies retrieved by the email report generator, ii) the authentication information for each received email, and iii) the report metadata that includes the information from the report sender as the organization name

and the period covered by the report. The ruf tag concerns failure (forensic) reports sent each time SPF and DKIM are not aligned. The fo tag can also provide other options for generating failure reports. Figure 1 shows an example: aggregate reports should be sent to rua@example.org and failure reports to ruf@example.com.

2.3.4 Role of the Reporting System. DMARC reports play a critical role in the email ecosystem: they help domain owners monitor and assess the efficacy of their DMARC policy. They also assist in refining anti-spoofing rules, email system configurations, identifying authentication setup issues, and preventing the unauthorized use of the domain for malicious purposes. For instance, based on DMARC reports, domain owners may observe that emails involved in phishing campaigns, claiming to be from their domain but originating from unauthorized servers, consistently fail DMARC checks. In response, they can then tighten their DMARC policy to reject or quarantine these emails, reducing the risk of abuse. Another example concerns DMARC reports that may reveal cases in which emails from a legitimate server fail SPF or DKIM checks due to misconfigurations such as an outdated SPF record that does not list all designated email service providers, leading to authentication failures for legitimate emails consequently rejected by receivers.

2.3.5 External Destination Verification (EDV). As noted in RFC 7489 and by Ashiq et al. [8], malicious actors could specify any email addresses in rua or ruf tags. DMARC reports will then be generated with respect to the domain name owned by the attacker and sent to the email address of the victim, which may result in a flooded mailbox. As a remediation, the report senders should verify that the domain name in the rua and ruf tag option to receive reports by performing the EDV. For instance, the DMARC record of the example.com domain in Figure 1 has the rua tag containing an email address with the example.org domain part that does not belong to the example.com organizational domain. Thus, to opt-in to receive DMARC reports for example.com, the domain owner of example.org should add a TXT DNS record located at example.com._report._dmarc.example.org with at least the ‘v=DMARC1’ tag. The external domain may specify a new rua or ruf tag within the same organizational domain. In this case, the report generator should send (or redirect) the report to the new URI (email address) specified in the rua or ruf tag. The reporters should proceed with the EDV before sending any DMARC reports.

3 Testing DMARC Operation of Email Service Providers

In this section, we report on testing results of the ESP operation with respect to the RFC 7489 specifications and recommendations.

To test the operation of the DMARC reporting system, we have registered email accounts at 25 ESPs that appeared in previous work
<table><caption>Table 1: Results of ESP testing that show their adherence to specifications.
X did not use or respect the feature, ✓ respects the feature, - feature cannot be evaluated</caption><thead><tr><th>Features</th><th>163.com</th><th>fastmail.com</th><th>freemail.hu</th><th>gmail.com</th><th>gmx.net</th><th>hcy.com</th><th>inbox.lv</th><th>interia.pl</th><th>kobahnow.com</th><th>laposte.net</th><th>mail.com</th><th>mail.ru</th><th>mailfence.com</th><th>naver.com</th><th>op.pl</th><th>orange.fr</th><th>outlook.com</th><th>protonmail.com</th><th>rediff.com</th><th>sapo.pt</th><th>seznam.cz</th><th>tutanola.com</th><th>yahoo.com</th><th>yandex.com</th><th>zohomail.eu</th></tr></thead><tbody><tr><td>F<sub>1</sub>: DMARC<sub>record</sub></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>F<sub>2</sub>: DMARC<sub>header</sub></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>F<sub>3</sub>: DMARC<sub>rua</sub></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>F<sub>4</sub>: DMARC<sub>ruf</sub></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>F<sub>5</sub>: EDV<sub>query</sub></td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>F<sub>6</sub>: EDV<sub>control</sub></td><td>-</td><td>✓</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>F<sub>7</sub>: EDV<sub>failure</sub></td><td>-</td><td>✓</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>F<sub>8</sub>: EDV<sub>overwrite</sub></td><td>-</td><td>✓</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr></tbody></table>

[9, 17, 18, 29], and others found during our analysis of DMARC reports. We have only selected free-of-charge ESPs, those with a trial period, and available for registration (without any geographic restrictions). To ensure representativeness, we have only considered one domain per ESP, even though some ESPs manage several domain names (e.g., **yahoo.com**, **aol.com**, **netscape.net** managed by Yahoo).

## 3.1 Methodology

The testing experiment involves sending emails to ESPs to trigger their anti-spoofing mechanisms. For each ESP, we configure a domain name with appropriate SPF, DKIM, and DMARC records. After sending emails, we retrieve all the emails from the mailboxes, analyze DNS queries to name servers and monitor the email address for the corresponding **rua** and **ruf** tags.

Since ESPs obtain DMARC records from our name servers, we analyze the deployment of anti-spoofing authentication and reporting mechanisms by monitoring the queried domains. For instance, for the **esp-ex** ESP, we send an email from the **esp-ex.example.com** subdomain and we monitor the queries for the **TXT** records of **_dmarc.esp-ex.example.com** (refer to the 'DNS query: SPF, DKIM, DMARC' step illustrated in Appendix, Figure 3). Using a unique subdomain per ESP effectively mitigates any side effects of DNS caching and potential shared resolver infrastructure.

Using name server logs and received emails, we collect data on a set of eight features defined as follows: $F_1$ (DMARC<sub>record,txt</sub>) verifies if the ESP queries the records related to the anti-spoofing mechanisms. $F_2$ (DMARC<sub>header</sub>) verifies the presence of the authentication information in the **Authentication-Results** [24] email headers. $F_3$ (DMARC<sub>rua</sub>) and $F_4$ (DMARC<sub>ruf</sub>) indicate whether the ESPs send the aggregate and failure reports. Features $F_5-8$ complete the behavior of ESPs related to the EDV and can only be evaluated if the ESP sends reports. $F_5$ (EDV<sub>query</sub>) indicates if the ESP performed the EDV, $F_6$ (EDV<sub>control</sub>) is a control feature: it verifies the delivery of report in external email addresses with a correctly set up infrastructure. $F_7$ (EDV<sub>failure</sub>) verifies if the ESP does not send the reports when the EDV fails. Finally, $F_8$ (EDV<sub>overwrite</sub>) verifies if reports are sent to the new email address present in the EDV record.

## 3.2 Results

Table 1 summarizes the results. While **fastmail.com** is the only ESP respecting all the features except sending failure reports, our analysis shows that only 2 ESPs did not request DMARC records ($F_1$). Among the 23 ESP querying the DMARC record, 11 of them do not provide DMARC authentication information in the **Authentication-Results** header, the information accessible to the final email recipient. Nevertheless, as noted by Hu et al. [17], it does not imply they do not use the anti-spoofing mechanism. For example, **seznam.cz** does not provide the DMARC outcome in **Authentication-Results** ($F_2$) but it sends the aggregate report ($F_3$) with the information.

While only 6 ESPs provide aggregated DMARC reports ($F_3$), none sent us failure reports ($F_4$). We also found that out of 5 ESPs that performed external verification ($F_5$), two of them did not send any DMARC report when the EDV failed ($F_7$). Even if the EDV is an optional feature, Ashiq et al. showed that it raises security issues [8].

Concerning **outlook.com**, we received aggregate reports when the monitoring address was within the organizational domain. However, no reports were received when the monitoring address fell outside the organizational domain. Finally, the majority of the ESPs, apart from **fastmail.com**, sent (redirected) the report to an overwritten email address ($F_8$). While this feature is recommended by RFC 7489, it is not mandatory for the report generator to apply the redirection.

## 3.3 Comparison with Prior Work

While our results are aligned with those of Ashiq et al. [8], who similarly found no ESP sending failure reports, it is noteworthy to highlight the different behaviors uncovered by their study compared to ours: their study showed that **163.com** proceeded with the EDV query and sent aggregate reports and **yahoo.com** did not undertake the EDV during their study period. However, our measurements reveal that **163.com** does not perform the EDV or send aggregate reports anymore. We have also found that **yahoo.com** performed the EDV DNS query. Note that Ashiq et al. communicated their findings to the respective organizations in January 2023. We performed our tests in September 2023, eight months after their interaction with ESPs, suggesting that Yahoo might be fixing the issue. Factors such as email thresholds, which may determine when
DMARC actions are triggered based on the email volume, were
not considered in our experiments. Our measurement datasets are
publicly available <fnref n="1" />

## 3.4 Discussion

Our study shows that ESPs minimally adopt DMARC reporting features. While most ESPs retrieve DMARC records, they generally refrain from providing DMARC reports: our testing campaign found that only 6 ESPs send aggregate reports, and none sends failure reports.

Discussions within the community regarding DMARC failure reporting [1, 31] and legal assessments [3, 4] suggest privacy concerns as the main reason, particularly the inclusion of sensitive information such as the original message.

While this reluctance underscores the need to protect user data, the lack of aggregate reports hampers the ability to fine-tune DMARC policies, improve email security, and ensure deliverability. Without this feedback, domain owners may miss early signs of phishing campaigns or other malicious activities, reducing their ability to respond promptly and mitigate threats.

# 4 Monitoring and Analyzing DMARC Aggregate Reports

To pinpoint the technical challenges and shortcomings faced by report receivers, we perform an analysis of a substantial volume (40,157) of aggregate reports received on an email address specified in the DMARC record at a prominent European university over a span of 19 months (April 2022 to November 2023). In this section, we first explore the origins of the reports, give the details of the syntactic issues, and conclude by presenting recommendations on effective report processing.

## 4.1 Who Sends the Reports?

To identify the entities within the DMARC reporting ecosystem, we have categorized the senders of reports into five groups: i) publicly available ESPs (e.g., Yahoo, Google), ii), Network and Security Solutions (NSS) as a core business company (e.g., Cisco, Amazon, Infomaniak.com) iii), other Companies that may provide emails to their customers (e.g., banks such as BNP Paribas or private companies such as haribo.com) iv), Universities v), other Public organizations, such as governments and hospitals, e.g., gov.au, Europa.eu, wales.nhs.uk, vi), and Others, including personal, NGO, and undefined. It is important to note that the distribution of reporters and their types may vary among monitored infrastructures. For example, a significant percentage of the received emails originate from universities in our study (17.56%), likely due to our monitoring of the university email infrastructure.

Figure 2 shows that although the ESP category has the fewest individual organizations (8), it contributes to 42.82% of the received reports. Moreover, a moderate number of organizations (43) in the Network and Security Solutions category generate a relatively large number of reports. Overall, two categories engaged in email services (8 and 43) generated 78.34% of the received emails.

<figure><img src="image_3.png" /><figcaption>Figure 2: Received and failure reports by organization category</figcaption></figure>

It is worth noting that some companies may delegate the management of their email systems to ESPs (e.g., companies using Yahoo services) or to companies in the Network and Security Solutions category. Nevertheless, companies, public organizations, and others collectively contribute to 21.65% of the senders, which indicates their commitment to security and active participation in the DMARC ecosystem.

## 4.2 Parsing the Subject of Aggregate Reports

RFC 7489 defines the email subject (`dmarc-subject`) in the Augmented Backus-Naur Form (ABNF). The subject contains three parts: i) the reported domain name, ii) the domain name that submits the report (the submitter), iii) and a unique ID. Using strict regular expressions [6]

(note that the expression is already permissive in comparison to the ABNF specification), we obtain 5,564 reports that either do not match or for which the matching results are not syntactically valid according to RFC 7489. For instance, some subjects do not respect the case-sensitiveness of the `dmarc-subject` specification. ('Report Domain:' and 'Submitter:'), the submitter value is not a valid domain name, or the 'Report-Id' field is wrongly formatted. Moreover, one of the submitters surrounds the information with curly brackets and provides additional 'Date:' information. To cope with these problems, we propose the following steps in the analysis:

* lowercase the subject,

* remove the following characters: `{`, `}`, `;`, `]`,

* replace the tab character (`\t`) by a whitespace,

* use the following regular expression [5]:
  `domain:([^\s]*[^\s]+).*submitter:([^\s]*[^\s]+).*` 
  `report-id:([^\s]*[^\s]+)`

With these steps, we still had problems with analyzing 406 reports because their subjects did not contain the required information, e.g., '[DMARC Report]< submitter_domain>!<domain>1650931200!1651017599'. In addition, we have obtained the subject information for 2,147 reports but the submitter is not a domain name.

<footnote n="1">¹https://doi.org/10.57745/IKYMKA</footnote>
## 4.3 Analyzing the Syntax of Aggregate Reports

RFC 7489 defines the XML format of aggregate reports: Appendix C of RFC 7489 specifies the XML Schema Definition (XSD) that defines the structure and different elements of aggregate reports.

We have analyzed the XML files contained in 40,157 reports according to the XSD schema provided in RFC 7489: 37,034 of them are invalid with 892,544 errors. The most common errors are the following:

* object `/record/auth_result/spf/scope` is missing in 22,927 reports,

* object `/record/identifiers/envelope_from` is missing in 22,242 reports,

* object `/version` is missing in 22,215 reports,

* object `/policy崇 published/fo` is missing in 21,667 reports,

* object `/record/auth_results/dkim` is at unexpected position in 4,372 reports,

* the element in object `/record/row/source/ip` does not match the right pattern in 1,867 reports.

We can attribute the high number of errors to the presence of other alternative XSD schemas for reports. RFC 7489 provides a link to the namespace<fnref n="2" /> pointing to an `.xsd` file different from the XSD definition in RFC 7489. As these two schema definitions (RFC 7489 and `dmarc.org`) are different, report processing leads to inconsistent results depending on the schema used by reporters. The use of the second schema explains the elements missing in reports: `scope`, `fo`, `envelope_from`, and `version` are not defined in the schema of RFC 7489. The error related to the `dkim` position comes from the XML definition of a sequence: the XML object `/record/auth_results/dkim` should be before `/record/auth_results/spf`. Finally, the `ip` related errors originate from address obfuscation—`X.X.X.X` is not a valid IP.

If we assume strict adherence to the formal definitions in RFC 7489, 14.37% of report subjects are poorly formatted, and 92.22% of reports do not conform to the XML schema definition, bringing the total of invalid reports to 94.93%. If 51 organizations responsible for sending the majority of aggregate reports fix both the subject and XML issues, the proportion of invalid reports would drop to 38.59%.

## 4.4 Discussion

Most aggregate reports cannot be correctly parsed based on the RFC 7489 specification. As a result, administrators without specific expertise either lose authentication information or must adapt their processes to handle empirical data, often struggling with varying interpretations. Domain owners may use third-party services (e.g., proofpoint.com or agari.com) for assistance. Our previous study [19] revealed that over 30% of DMARC records contain email addresses of the top five DMARC third parties. This outsourcing raises two main issues: data exfiltration to vendors, raising privacy and sovereignty concerns, and potential dependency on these services, leading to vendor lock-in and reliance on external providers.

## 5 Related Work

Several studies investigated DMARC adoption in the domain name ecosystem [8, 11–13, 15, 18, 19, 26–28, 30, 32–34].

Much work investigated the behavior of ESPs with respect to anti-spoofing mechanisms [8, 10, 12–14, 17, 18, 20, 29, 33]. Some of the studies [9, 12, 14, 17, 18, 29] aimed to determine the extent to which ESPs implement anti-spoofing mechanisms and why they are critical stakeholders in combating email spoofing. Ashiq et al. [8] performed a measurement study of the DMARC reporting deployment and management by investigating 7 popular ESPs and email hosting services. While their focus was on security implications of EDV non-compliancy, they only investigated actors that supported sending aggregate reports. Our tests offer a broader perspective by examining the ESP support (or the lack of it). We have expanded our testing scope from 7 to 25 ESPs including key actors such as outlook.com, mail.ru, mail.com, and seznam.cz. Additionally, we identify differences in results and explore potential reasons for these variations.

Besides these differences, our work aims to analyze the technical challenges specific to DMARC report processing. To the best of our knowledge, only a handful of studies explored this aspect [21, 23]. Kanako et al. identified unreliable DMARC reports and proposed a reporter rating approach that relies on DMARC feedback data [21]. Konno et al. used DMARC report clustering to distinguish legitimate email senders and identify valid email transactions, incorrectly marked as failing DMARC authentication [23]. We are the first to assess the syntactic structure of the DMARC reports and their suitability for automated processing.

## 6 Ways of Improvement

The IETF originally published DMARC RFC 7489 as an Informational document in March 2015. Since 2020, the DMARC Working Group (WG) has been developing the 'dmarcbis' draft. The latest version of the draft [16] aims to elevate DMARC to the Standards Track, obsoleting RFC 7489. Aggregate and failure reports will also have their own RFCs. Interestingly, the new specification mandates the EDV as a 'MUST' instead of a 'SHOULD'. We have participated in discussions in the WG regarding XML ambiguities and proposed three solutions (some of them have already been well received):

1. Develop and standardize an extended XML schema to ensure interoperability across all versions to improve automatic processing of all aggregate report versions as demonstrated by the TA-dmarc project [2].

2. Consider a JSON format for aggregated reports. The JSON simpler syntax facilitates automatic processing of reports and presents the advantage of a flexible structure. The JSON format is not currently under consideration, however, it may be in the future. Originally, the dmarcbis specification stated that "the aggregate reports are XML documents." However, following the discussions with us, this requirement has been removed.

3. Specify the format version in the aggregate report XML schema to avoid confusion. The IANA will register specific URIs for the new XML schema.

After long discussions in the WG about a bump to `v=DMARC2` (based on other arguments than the problems with the reporting system), the consensus was against the version bump. However,

<footnote n="2">²https://dmarc.org/dmarc-xml/0.1</footnote>
without the bump version, there might be reluctance among actors to enhance the reporting system. Moreover, the issue with multiple XSD definitions will likely persist. More generally, even if the reporting formats are extended and standardized, ESPs may lack incentives to send reports.

# 7 Ethical considerations

Our study does not raise ethical concerns. We performed the spoofing experiment by impersonating the domain names we registered specifically for this study, and to avoid unsolicited emails to ESPs, we have only sent emails to our own registered accounts. Additionally, our university partner only provided aggregate reports rather than failure reports due to potential privacy concerns, such as email content failing authentication, which could be included as attachments.

# 8 Conclusions

Our results show that only a minority of actors actively participate in DMARC reporting. Even those who participate do not follow the IETF specifications, leading to potential issues for domain owners attempting to deploy DMARC or improve its operation. Several major ESPs have the potential to drive significant changes in the reporting system. Our analysis of 40,157 aggregate reports reveals that as many as 94.93% contain incorrectly formatted report subjects or do not conform to the RFC-specific XML schema.

# Acknowledgments

We thank Simon Fernandez for his valuable feedback on the paper and the University Grenoble Alpes for providing us with the DMARC aggregate reports. This work has been partially supported by Grenoble Alpes Cybersecurity Institute (under the contract ANR-15-IDEX-02), the French Ministry of Research projects PERSYVAL-Lab under contract ANR-11-LABX-0025-01 and DiNS under contract ANR-19-CE25-0009-01.

# References

[1] 2016. Re: [dmarc-discuss] DMARC forensic reporting options. https://www.mail-archive.com/dmarc-discuss@dmarc.org/msg02595.html. (Accessed on 11/27/2023).

[2] 2017. GitHub - jorritfolmer/TA-dmarc: Add-on for ingesting DMARC aggregate reports into Splunk - github.com. https://github.com/jorritfolmer/TA-dmarc. [Accessed on 11/27/2023].

[3] 2018. GDPR's Impact on DMARC Data Collection. https://dmarcian.com/gdprsm-impact-on-dmarc-data-collection/

[4] 2018. Report on Compliance of DMARC with the EU GPR. https://certified-senders.org/wp-content/uploads/2018/08/Report_DMARC_and_GDPR.pdf

[5] 2023. Regex101; DMARC reports subject relaxed regular expression. https://regex101.com/r/2VDIyo/1 (Accessed on 06/14/2024).

[6] 2023. Regex101; DMARC reports subject strict regular expression. https://regex101.com/r/2VDIyo/2 (Accessed on 06/14/2024).

[7] E. Allman, J. Fenton, M. Delany, and J. Levine. 2009. DomainKeys Identified Mail (DKIM) Author Domain Signing Practices (ADSP). RFC 5617. RFC Editor.

[8] Md. Ishtiaq Ashiq, Weitong Li, Tobias Fiebig, and Taejoong Chung. 2023. You've Got Report: Measurement and Security Implications of DMARC Reporting. In 32nd USENIX Security Symposium (USENIX Security 23). USENIX Association, Anaheim, CA, 4123-4137. https://www.usenix.org/conference/usenixsecurity23/presentation/ashiq

[9] Birk Blechschmidt and Ben Stock. 2023. Extended Hell(o): A Comprehensive Large-Scale Study on Email Confidentiality and Integrity Mechanisms in the Wild. In 32nd USENIX Security Symposium. USENIX Association, 4895-4912.

[10] Jianjun Chen, Vern Paxson, and Jian Jiang. 2020. Composition Kills: A Case Study of Email Sender Authentication. In 29th USENIX Security Symposium. 2183-2199.

[11] Stefan Czybik, Micha Horlboge, and Konrad Rieck. 2023. Lazy Gatekeepers: A Large-Scale Study on SPF Configuration in the Wild. In Proceedings of the 2023 ACM on Internet Measurement Conference (IMC '23). ACM, 344-355.

[12] Casey Decchio, Tarun Yadav, Nathaniel Bennett, Alden Hilton, Michael Howe, Tanner Norton, Jacob Rohde, Eunice Tan, and Bradley Taylor. 2021. Measuring Email Sender Validation in the Wild. In Proceedings of the 17th International Conference on Emerging Networking Experiments and Technologies (CoNEXT). ACM, 230-242.

[13] Zakir Durumeric, David Adrian, Ariana Mirian, James Kasten, Elie Bursztein, Nicolas Lidzborski, Kurt Thomas, Vijay Eranti, Michael Bailey, and J. Alex Halderman. 2015. Neither Snow Nor Rain Nor MITM...: An Empirical Analysis of Email Delivery Security. In Proceedings of the 2015 IMC. ACM, 27-39.

[14] Ian D. Foster, Jon Larson, Max Masich, Alex C. Snoeren, Stefan Savage, and Kirill Levchenko. 2015. Security by Any Other Name: On the Effectiveness of Provider-Based Email Security. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 450-464.

[15] Ivan Gojmerac, Patrick Zwickl, Gabriel Kovacs, and Christoph Steindl. 2015. Large-scale active measurements of DNS entries related to e-mail system security. In 2015 IEEE International Conference on Communications (ICC). 7426-7432.

[16] Todd Herr and John R. Levine. 2024. Domain-based Message Authentication, Reporting, and Conformance (DMARC). Internet-Draft draft-ietf-dmarc-dmarcbis-31. Internet Engineering Task Force. https://datatracker.ietf.org/doc/draft-ietf-dmarc-dmarcbis/31/ Work in Progress.

[17] Hang Hu and Gang Wang. 2018. End-to-End Measurements of Email Spoofing Attacks. In 27th USENIX Security Symposium. USENIX Association, 1095-1112.

[18] Hang Hu and Gang Wang. 2018. Revisiting Email Spoofing Attacks. CoRR abs/1801.00853 (2018). arXiv:1801.00853 http://arxiv.org/abs/1801.00853

[19] Olivier Hureau, Jan Bayer, Andrzej Duda, and Maciej Korczyński. 2024. Spoofed Emails: An Analysis of the Issues Hindering a Larger Deployment of DMARC. Springer Nature Switzerland, 232-261.

[20] Georgios Kambourakis, Gerard Draper Gil, and Ignacio Sanchez. 2020. What Email Servers Can Tell to Johnny: An Empirical Study of Provider-to-Provider Email Security. IEEE Access 8 (2020), 130066-130081.

[21] Konno Kanako, Kitagawa Naoya, Yamai Nariyoshi, and Yamai Nariyoshi. 2017. DMARC Reporter Rating Method by Massive Reports Analysis, Vol. 117. 83-88.

[22] S. Kitterman. 2014. Sender Policy Framework (SPF) for Authorizing Use of Domains in Email, Version 1. RFC 7208. RFC Editor.

[23] Kanako Konno, Naoya Kitagawa, and Nariyoshi Yamai. 2020. False Positive Detection in Sender Domain Authentication by DMARC Report Analysis. In Proceedings of the 3rd International Conference on Information Science and Systems (ICISS '20). ACM, 38-42.

[24] Murray Kucherawy. 2019. Message Header Field for Indicating Message Authentication Status. RFC 8601. https://doi.org/10.17487/RFC8601

[25] M. Kucherawy and E. Zwicky. 2015. Domain-based Message Authentication, Reporting, and Conformance (DMARC). RFC 7489. RFC Editor.

[26] Sourena Maroofi, Maciej Korczyński, Arnold Hölzel, and Andrzej Duda. 2021. Adoption of Email Anti-Spoofing Schemes: A Large Scale Analysis. IEEE Transactions on Network and Service Management 18, 3 (2021), 3184-3196.

[27] Yevheniya Nosyk, Olivier Hureau, Simon Fernandez, Andrzej Duda, and Maciej Korczyński. 2023. Unveiling the Weak Links: Exploring DNS Infrastructure Vulnerabilities and Fortifying Defenses. In 2023 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW). IEEE Computer Society, 546-557.

[28] Adam Portier, Henry Carter, and Charles Lever. 2019. Security in Plain TXT. In Detection of Intrusions and Malware, and Vulnerability Assessment (DIMVA). Springer International Publishing, 374-395.

[29] Kaiwen Shen, Chuhan Wang, Minglei Guo, Xiaofeng Zheng, Chaoyi Lu, Baojun Liu, Yuxuan Zhao, Shuang Hao, Haixin Duan, Qingfeng Pan, and Min Yang. 2021. Weak Links in Authentication Chains: A Large-scale Analysis of Email Sender Spoofing Attacks. In 30th USENIX Security Symposium. 3201-3217.

[30] Pawel Szalachowski and Adrian Perrig. 2017. Short Paper: On Deployment of DNS-Based Security Enhancements. In Financial Cryptography and Data Security, Aggelos Kiayias (Ed.). Springer International Publishing, 424-433.

[31] Steven M Jones via dmarc discuss. 2019. Re: [DMARC-discuss] DMARC RUF. https://www.mail-archive.com/dmarc-discuss@dmarc.org/msg03144.html

[32] Chuhan Wang, Kaiwen Shen, Minglei Guo, Yuxuan Zhao, Mingming Zhang, Jianjun Chen, Baojun Liu, Xiaofeng Zheng, Haixin Duan, Yanzhong Lin, and Qingfeng Pan. 2022. A Large-scale and Longitudinal Measurement Study of DKIM Deployment. In 31st USENIX Security Symposium (USENIX Security 22). USENIX Association, Boston, MA, 1185-1201. https://www.usenix.org/conference/usenixsecurity22/presentation/wang-chuhan

[33] Chenkai Wang and Gang Wang. 2022. Revisiting Email Forwarding Security under the Authenticated Received Chain Protocol. In Proceedings of the ACM Web Conference (WWW). ACM, 681-689.

[34] Masanori Yajima, Daiki Chiba, Yoshiro Yoneya, and Tatsuya Mori. 2023. A First Look at Brand Indicators for Message Identification (BIMI). In International Conference on Passive and Active Network Measurement. Springer, 479-495.
<figure><img src="image_4.png" /><figcaption>Figure 3: Examples of DKIM, SPF, and DMARC used in the email ecosystem.</figcaption></figure>

# A Appendix

Figure 3 illustrates the email ecosystem with an example of the entities involved in the DKIM, SPF, and DMARC processing: a domain owner and an email sender (**Paypal.com**), an email receiver (**Yahoo.com**), and an external destination for processing DMARC reports (**Agari.com**). Alice sends an email with the **From:** header of **alice@Paypal.com** to **bob@yahoo.com**. The MDA of **yahoo.com** queries the name server of **Paypal.com** to retrieve the anti-spoofing policies set by the domain owner (DKIM, SPF, and DMARC records). When the MDA performs the DMARC check, the Alice email results in **pass** because her MTA sender IP address corresponds to the SPF **ip4** mechanism. The email spoofed by Mallory does not

pass the SPF verification and results in **reject** according to the DMARC policy of **Paypal.com**. In this case, the policy advises the MDA to send a DMARC forensic report to **Agari.com**, the company chosen by **Paypal.com** to handle DMARC reports, specified in the ** rua** and **ruf** tags. Thus, the MDA performs the external validation of the **Agari.com** email address, verifies the corresponding **Paypal.com._report._dmarc.ruf.Agari.com** TXT record at the **Agari.com** name server, and sends the DMARC forensic report to **d@Agari.com**. Within 24 hours, Yahoo sends an aggregate report to **d@Rua.Agari.com**, which contains the authentication information of the Alice and Mallory emails. Based on the DMARC reports, DMARC experts may suggest improvements to the policies configured by **Paypal.com**.
# B Artifact Appendix

## B.1 Abstract

The testing experiment described in Section 3 measures the behavior of 25 Email Service Providers (ESPs) towards the DMARC reporting system. The artifacts include data collected through active and passive measurements and the code used to generate Table 1: "Results of ESP testing that show their adherence to specifications."

## B.2 Artifact check-list (meta-information)

Obligatory. Use just a few informal keywords in all fields applicable to your artifacts and remove the rest. This information is needed to find appropriate reviewers and gradually unify artifact meta information in Digital Libraries.

* Data set: Raw email, DNS Zonefile, Rspamd logging file, Bind9 logging file

* Run-time environment: Python3.8 virtual environment on Linux OS.

* Execution: Jupyter Notebook

* Output: LATEXtabular

* How much time is needed to prepare workflow (approximately)?: ≤ 1 minute

* How much time is needed to complete experiments (approximately)?: ≤ 1 minute

* Publicly available?: Yes

* Code licenses: CC0 1.0 Universal

* Data licenses: etalab 2.0

* Archived : Data : https://doi.org/10.57745/IKYMKA, Code (Software heritage) : https://archive.softwareheritage.org/swh:1:dir:c23b067e725c77e4486c4cee3725087a6fd23803;origin=https://github.com/drakkar-lig/dmarc-conext24;visit=swh:1:snp:767000a72f2eb0de4c54aa1274181c8a0236daed;anchor=swh:1:rev:99a8cdcaad1b35b5d368a064b90f9f0fff58d516

## B.3 Description

Measurements were performed on a server running Ubuntu 22.04.3 LTS with IPv4 and IPv6 connectivity. The main services used for the experiment are Bind9 and the mail suite Mailcow.

We have sent a total of 16 emails to the email addresses associated with the registered account of 25 ESP. Those 16 emails were sent using 4 different domain names. For each domain, we have emails with DKIM alignments and SPF alignments following the cartesian products:

`{alignedDKIM, notalignedDKIM}×{alignedSPF, notalignedSPF}`

The data repository contains the email present in the ESP inboxes, the Zonefile (RFC 1035) of the 4 domain names used to send the emails, the Zonefile of the external domain name subject to the DMARC External Destination Verification (EDV), the Bind9 log files containing the DNS queries sent to our authoritative nameserver, the aggregate report we have received, and the log from the Rspamd services running in the Mailcow suite.

We have used the server logs and the emails to evaluate the 8 features of Table 1. The GitHub repository contains a Jupyter Notebook to generate the LATEXcode of Table 1. The execution of the Notebook

also provides subsidiary files which we found interesting to keep as artifacts for future research works.

More descriptions can be found in the 'README.md' file of the code and the data repository.

### B.3.1 How to access.
Data sets are available in the data repository: https://doi.org/10.57745/IKYMKA.
The code to generate the table is available at https://github.com/drakkar-lig/dmarc-conext24

Everything can be retrieved using the commands:

```
git clone https://github.com/drakkar-lig/dmarc-conext24
cd dmarc-conext24
wget --content-disposition \ 'https://entrepot.recherche.data.gouv.fr/api/'\ 'access/datafile/:persistentId?persistentId='\ 'doi:10.57745/GXFEMF' ; unzip dmarc-datasets-conext24.zip ;
```

The artifact occupied 11.8 MiB disk space after unpacking and code execution.

### B.3.2 Software dependencies.
The code has been produced and run on a Unix-Path-like environment. The installation procedure is written for an Ubuntu environment and the following packages must be installed: python3.8, python3.8-venv, python3-pip. All the Python dependencies are listed in the file 'requirements.txt'. The PDF is generated using pdflatex.

## B.4 Installation

In the folder created previously, run the following command:

```
python3.8 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt
```

## B.5 Evaluation and expected results

The results can be found in the data repository. The results can be re-regenerated by running a Jupyter Notebook server and executing the Notebook 'GenerateData.ipynb'

You may also run the following command:
`jupytertext --execute GenerateData.ipynb`

A 'results' folder will be created and contain the LATEXtable in the file 'dmarc_features_table.tex'

The file 'generated_latex_document.tex' is a standalone LATEXdocument which includes the dmarc_features_table. You may compile the document as follows:

```
cd results/
pdflatex generated_LaTeX_document.tex
```

The command above results in the creation of the file 'generated_LaTeX_document.pdf'

## B.6 Notes

Even though it does not impact the measurement analysis of the current paper, the content of the email titled "- SPAM orangefr Email Research Testing Dmarc configuration - SPK OK - DKIM NOK- 13 42 16.416362" is lost.