Automatic image cropping
By: Justin Van Zanten
Tags: Image Scaling, Content Automation, Kotlin, Ecommerce, Web Development

TL;DR — We fit an image to the requested aspect ratio by cropping them while trying to keep a user-selected point in the centre of the cropped image.

Our editors perform a lot of editing on pictures to make them fit the sizes of the images used on our site. In order to reduce to amount of time they had to spend, we tried to automate the process.

This article consists of a business and a technical part. The former will describe how we got to the solution we found and the latter will give a general way of implementing this solution. If you are only interested in the code you can skip the first half.

Example of dynamic cropping. The red circle indicates the selected focus point.

The Business case

All solutions originate from a problem. And knowing about a problem starts with an observation. Our observation was the fact that editors were spending a lot of time adding patches of colour as extra padding to images. They were not able to use the image provided to them without significant retouching. This prompted us to investigate the process.

The reason our editors had to change images is the fact that we use adaptive images on our site. We have loads of differently proportioned images for distinct screen sizes. Each component might have an image which will be cropped to multiple aspect ratios and the picture needs to look great on each of them.

Why can’t we just keep the same ratio?

Can we not design the image’s aspect ratio to be the same on all breakpoints? Unfortunately, this was not an option. A good example to illustrate why this was not feasible is our hero banner.

Our hero banner was designed to take advantage of the full with of the screen our customers use to view the website. This is in order to capture the user’s attention.

If the image has a large height compared to its width it will push a large portion of the remaining content on the site under the fold on wider screens. When you give the image a small height compared to its height the image will become rather small on narrow screens.

High image pressing remaining content under the fold (left) and a wide image becoming small (right).

We accepted handling the different aspect ratios as a requirement. So the task became: “Make sure that the images are automatically scaled no matter what picture sizes are uploaded.”

Our front end developers went to work with a possible solution. Their solution hides the sides of the picture which are too large to be included in the desired aspect ration. This appeared to be working for our editors.

But what about the shoes?!

After working a while it came to our attention that editors were again doing a lot of manual cropping and padding. It turned out that being a fashion store we had a lot of marketing material for shoes. The automatic cropping we created tended to cut away the most important part of the pictures which purpose it was to show off shoes.

Missing shoes on an image centred around shoes.

To combat this our editors proposed to enable the upload of multiple pictures in order to allow them to have full control over the cropping of the images. This would set us back to square zero as it would involve them to go back to cropping and padding manually again.

The final solution we came up with was to allow them to select a focus point. When a picture is too wide it would be cropped from the sides, from the top and bottom if it was too high. The cropping would be done in such a way that the focus point would be kept in the centre as much as possible.

A source image (top) has a focus point indicated by the red circle. The image is automatically cropped to a banner (bottom-left) or a square (bottom-right)

This solution still has some drawbacks. One requirement for this to work is editors uploading pictures which have a resolution large enough to be cropped and still maintain high quality. Because of the focus point gravitating towards the centre, the composition of the original picture might be compromised.

Recap of all the drawbacks of the various solutions.

The Code

Now the technical side. The first part is about the algorithm to crop the image. Following this is some code to select the focus point on an image.

The examples are written in Kotlin but are supported by pseudo code to allow you to create an implementation in other languages. The original code was written in Java.

The examples are intended to be implemented on any system. We used a servlet to make this available in our cms, but you could use the image scaler in an app as well.

Image Resizing

First, we take an image and some parameters which tell us how we want to transform the image. There will be one object responsible for this task. We will call this a service. This concept tends to lend itself to most frameworks really well.

Aside from the image, it will need the dimensions of the desired image’s size and the location of the focus point on the image. This location will be described with two floating number between 0 and 1. The former being the vertical component, 0 being the top and 1 the bottom, and the latter being the horizontal component, 0 the left and 1 the right of the image.

This is what the interface of the service looks like:

The implementation of the cropping algorithm starts by checking if the inputs are valid. This might save some unexpected errors. The desired width and height need to be at least 1 and the focus location needs to be between and 0 and 1 inclusive.

In this example, we will be transforming a 500 x 500 image to an 800 x 400 image. The focus point will be on 250 x 200 on the original image, Resulting in top: 0.4 and left: 0.5 respectively.

The image needs to be cropped in order to get the desired ratio. To do this we need to cut away the largest sides when compared to the desired image. We can decide which side this is by dividing the original width by the desired width and the original height by the desired height. Whichever side has the highest ratio will be the side we cut.

From this, we get a width ratio of 0.625 and a height ratio of 1.25. Thus the cut needs to come from the top and bottom.

We can use the calculated ratio to determine the remaining size of the original image we want to keep. The size of the side of the area we want to keep is the desired side’s size multiplied by the other side’s ratio. The other side will keep its original size.

400 * 0.625 = 250. The area of the image we want to keep has a size of 500 (original image’s width) by 250 (calculated height).

The location of the gravity point is relative to the size of the original image. In order to get the location of the selected point in absolute pixels, we multiply the side’s size by the related parameter.

The gravity points original vertical location was 500 * 0.4 =200.

There will be a certain distance cut away from the border of the image. This might be difficult to figure out but it needs to be the absolute location of the gravity point minus half of the calculated width. Unless you encounter the border of the image before that.

The top cut will be on 200 - 250/2 = 75. To make sure it is not over the edges of the image it needs to be > 0 and < 250.

In order to cut this calculated area out of the original image, we utilize the buffered image’s getSubimage functionality. This cropped image now needs to be scaled to the correct size. Because there are libraries better suited for this than our own handy craft we use those. In this case Scalr, which will also lend its name to the implementation.

Showcase

Seeing is believing. How do we show this? We can use a framework in order to show images quickly. Here we use tornadofx. This app can show an image.

Okay, this shows an image. We want to see the results. Here is the same kind of app but with a transformed image.

You can play around with the parameters and the image yourself. Here is an app which allows you to select the parameters without changing the code.

For the full code, you can check https://github.com/JustinvanZanten/resize