# The Big Picture: An Improved Method for Mapping Shipping Activities
## 3 Generating Density Maps from Raw AIS Data
### Data Cleaning  
As raw AIS messages have different types, static and dynamic information should be coupled manually. Information regarding the type and class of the vessel is extracted by 'Static and Voyage Related Data' messages and merged with the dynamic (positional) messages for the corresponding vessel, resulting in a more complete AIS dataset. Furthermore, removing erroneous and unnecessary messages from AIS datasets is a crucial component for any further analysis of maritime movement. In this step, we clean the data through a series of filters on AIS messages, in order to smooth the noise and potentially decrease the error in their measurements. More precisely, the filters applied to each input message are the following:  
Figure 2: Flowchart for extracting density maps from raw AIS, by decoding and cleaning the data and calculating the corresponding density. Improving the AIS dataset through the gap filling mechanism or fusing the data with satellite image detections (orange) may be incorporated for more accurate results. 1. _Empty fields_: messages that monitor movement, like the AIS messages, may include a plethora of features. Besides primary features (positional and temporal) that denote the exact position of the moving vessel, other fields regarding its characteristics or its current state are usually provided. For the purpose of effectively analyzing the input data, we require that each positional message includes non-empty values in the following fields: * Positional features: Vessel Longitude, Vessel Latitude, Timestamp of AIS occurrence (expressed in UNIX time in milliseconds). * Movement measurements: Vessel Speed-over-Ground (SoG-measured in knots) and Vessel Course-over-Ground (CoG-measured in degrees). 2. _Invalid movement fields_: While most messages include the fields regarding a vessel's movement (SoG, CoG), in some instances, these fields carry invalid values. In such cases, the messages are characterized as erroneous and are discarded. The thresholds indicating valid movement are as follows: * Speed-over-Ground: a real number between 0 and 80 knots. * Course-over-Ground: a real number between 0 and 360 degrees. 3. _Invalid vessel identification number_: With each AIS message referring to a single vessel, a field dedicated for its identification is needed. The Maritime Mobile Service Identity (MMSI) convention is widely used while referring to AIS transmitters (i.e., vessels) [11], with each single entry being a series of nine characters. Messages with a shorter or longer MMSI length, as well as messages whose MMSI falls into some exception values (123456789, 0.12345, 00000000, 11111111, et al. ), are discarded. 4. _Areas of interest/Land-masking_: While our approach may be applied regardless of the area in question, defining the space of reference is a crucial part for moving forward for two reasons: * Removing data that refer to areas outside of the purpose of the execution scenario. * Removing data that include erroneous coordinates, i.e., not valid longitude/latitude or points on land. 5. _Down-sampling_: Although the frequency where each vessel is transmitting a positional message is usually desired to be as high as possible, having too many messages may result in considerable delays while processing. In order to overcome this issue, a down-sampling is performed upon the input data. The question at this point is whether we are able to disregard some sample points without sacrificing the quality of the trajectory data required for the target application. For this purpose, the trajectories are filtered so that consecutive messages from the same vessel have at least \(k\) minutes between them, which also remove all duplicate messages as a result. 6. _Time-frame_: Restricting the time-frame where AIS messages are to be included in the end result may be useful for creating a custom dataset for analysis and removing messages with erroneous timestamps, due to noise.