Selective request-payload logging in spring
By: Patrick Adler
Tags: Passwords, Json, Spring, Logging, Requestlogging

A project I used to work on used Keycloak in the background, but routed everything through the Middleware.

Login and registration were implemented by the Front-End sending DTOs to the Middleware which used the admin-client to perform logins and registrations (including verification emails and everything else Keycloak cloud do for you out-of-the-box).

One day I was looking at the logs, more or less randomly because I suspected there could have been a database outage and just stumbled across an entry like this:

DEBUG o.p.m.s.c.RequestLoggingFilter — After request [POST /login, payload={

“password”: “password1”,

“username”: “test”

}]

WE WERE LOGGING CLEAR TEXT PASSWORDS!

I immediately checked our DTOs. The password fields were annotated with @JsonIgnore and @JsonProperty(access = JsonProperty.Access.READ_ONLY) so everything should be alright, shouldn’t it?

I googled “@JsonProperty(access = JsonProperty.Access.READ_ONLY)” not working and “@JsonIgnore not working” but with no success.

Some time ago we had replaced our custom logging filter with spring boot’s out-of-the-box solution:

The problem is that this filter is so far outside in the filter chain, that it does not know anything about the request payload and hence cannot omit fields according to our annotations.

The first solution v1

Thankfully, The integration of the filter was already prepared for a quick fix of the problem until a better solution can be implemented. It looked something like this:

The default filter was extended with a List of URIs to exclude from logging, excluding the /actuator/health endpoint by default. The configuration could be adjusted in the application.yaml.

I quickly added this to the application.yaml and ensured a quick deployment:

api.logging.urisToExclude: "/actuator/health,/login,/register"

The treacherous log messages were gone and I called it a day.

I was sure we could easily improve the RequestLoggingFilter to take a list of URIs to exclude the payload from.

But I wanted more, I wanted to try and omit only the sensitive data from the logged payload.

The second solution v2

First, I implemented the simple solution, excluding the payload for certain URIs:

The log now looked like this:

DEBUG o.p.m.s.c.RequestLoggingFilter — After request [POST /login]

v2.5

Note that this solution might not be thread-safe. The “setIncludePayload” property is for the entire Filter until the next request comes in.

This could affect other requests whose payload-message is being right now.

A better solution is not much more verbose:

This will still log the request itself but omit the entire payload. This is ok for our use-case but I still wanted to see what’s possible.

The complex, maybe crazy solution v3

To be able to exclude the fields annotated with @JsonIgnore, we need to identify the target class of the request body.

I quickly found the RequestMappingHandlerMapping class which helped me in the process, but it did not contain everything I needed so there was much work to do.

I iterated the handler methods and created a map with the HTTP method as a key and maps with corresponding handler methods as value. The second map has the patterns as its key and the handler methods as value.

When a request comes in, the overridden “getMessagePayload” is called.

The only modification is that it calls “censorPayload” on the message payload.

This retrieves the handler methods for the request method, and then uses RequestMappingHandlerMapping.match to find the matching pattern, resulting in the handler method for this request.

This handler method is then inspected for the parameter annotated with “RequestBody”. This is our target class.

Now the payload is deserialized with the object mapper and immediately serialized again, to leverage its support of @JsonIgnore, etc.

The resulting string is returned as payload.

this implementation could be done differently, more on that later.

Result:

DEBUG o.p.m.s.c.RequestLoggingFilter — After request [POST /users/login, payload={“username”:”test”}]

Great Success! This is what we wanted all the time. Pretty printing is trivial. Just replace

return mapper.writeValueAsString(o);

with:

return mapper.writerWithDefaultPrettyPrinter().writeValueAsString(o);

Result:

DEBUG o.p.m.s.c.RequestLoggingFilter — After request [POST /users/login, payload={

“username” : “test”

}]

The implementation of censorPayload with the object mapper is “straightforward” or in other words LAZY and inefficient.

A custom implementation could be much better, but the complexity is high.

The field annotations in the DTO have to be analyzed and the corresponding substrings removed from the payload string.

It might even be just as inefficient and slow.

Measuring the actual performance impact is a future task. I might make updates to this article with the findings and a potentially better implementation. Maybe I can replace the password with ***.

However, it is unclear whether we will actually be using this or just keep it simple by excluding the entire payload. So far this is more of a demonstration or proof of concept.

I’m not saying it is a really good solution as it feels a little weird.

Maybe I was so preoccupied with whether or not I could, I didn’t stop to think if I should.

Conclusion

It is possible to use the existing annotations and exclude unwanted fields from logging. However, it is quite complex and should be thought through well.

I feel less bad about it since I know that it can happen to anybody, including Twitter and Github.