A method of processing video data, comprising: determining, for a conversion between a current block of a video and a bitstream of the video, that one or more above-left neighboring samples outside the current block are unavailable for a coding tool applied to the current block, wherein the coding tool comprises an adaptive loop filter process, wherein the one or more above-left neighboring samples are unavailable when the one or more above-left neighboring samples are located in a different video unit from the current block and the coding tool using samples across the video unit is disallowed; padding the one or more above-left neighboring samples with above neighboring samples outside the current block, in case that left neighboring samples outside the current block are available and the above neighboring samples are available; and performing the conversion based on the padded one or more above-left neighboring samples, wherein the padded one or more above-left neighboring samples are used for a classification operation in the adaptive loop filter process, wherein the padded one or more above-left neighboring samples are used to determine a classification filter index in the classification operation, and the classification filter index is utilized to determine a filtering coefficient set, wherein during the classification operation, a classification of an MxN region of the current block is determined based on gradients of a subset of samples of a PxQ region including the MxN region, wherein the current block is a coding tree block, where the MxN region is a 4*4 region and the PxQ region is an 8*8 region, and wherein when there is a sample in a 4×4 block located at a left boundary of the current block, left-most two columns of samples in an 8×8 region are padded, or when there is a sample in a 4×4 block located at a top boundary of the current block, top-most two lines of samples in a 8×8 region are padded. determining, for a conversion between a current block of a video and a bitstream of the video, that one or more above-left neighboring samples outside the current block are unavailable for a coding tool applied to the current block, wherein the coding tool comprises an adaptive loop filter process, wherein the one or more above-left neighboring samples are unavailable when the one or more above-left neighboring samples are located in a different video unit from the current block and the coding tool using samples across the video unit is disallowed; padding the one or more above-left neighboring samples with above neighboring samples outside the current block, in case that left neighboring samples outside the current block are available and the above neighboring samples are available; and performing the conversion based on the padded one or more above-left neighboring samples, wherein the padded one or more above-left neighboring samples are used for a classification operation in the adaptive loop filter process, wherein the padded one or more above-left neighboring samples are used to determine a classification filter index in the classification operation, and the classification filter index is utilized to determine a filtering coefficient set, wherein during the classification operation, a classification of an MxN region of the current block is determined based on gradients of a subset of samples of a PxQ region including the MxN region, wherein the current block is a coding tree block, where the MxN region is a 4*4 region and the PxQ region is an 8*8 region, and wherein when there is a sample in a 4×4 block located at a left boundary of the current block, left-most two columns of samples in an 8×8 region are padded, or when there is a sample in a 4×4 block located at a top boundary of the current block, top-most two lines of samples in a 8×8 region are padded.