Handling outliers during data cleaning involves techniques such as Winsorization, which replaces extreme values with the nearest non-outlier value, and Z-score, which identifies outliers based on their deviation from the mean and standard deviation. Pandas fillna method can be used in combination with these techniques to handle outliers effectively.