A non-transitory computer readable media for tissue characterization in medical imaging, the non-transitory computer readable media comprising instructions which, when executed by a computer, cause the computer to carry out at least the following steps: receiving scanning data of a patient scanned with a medical scanner, the scanning data providing multiple frames of data representing a tissue region of interest in the patient; extracting, by a processor, values for features from the frames of data; classifying, by a machine-learnt classifier implemented by the processor, a therapy response of the tissue of the tissue region from the values of the features as input to the machine-learnt classifier; wherein extracting and classifying comprise extracting and classifying based on Siamese convolutional networks comprising two parallel networks which process pre and post treatment data, wherein the Siamese convolutional networks are trained to extract features from given multi-modal, multi-dimensional images, IM1 and IM2, which are used as input images, wherein one branch of the Siamese convolutional network is used to input IM1 representing pre-treatment and another branch is used to input IM2 representing post-treatment, wherein relevant features are automatically determined as part of training, wherein the two parallel networks are trained with exactly the same parameters, and the Siamese convolutional networks are linked so that the same weights for the parameters defining the networks are used in both branches, wherein classifying the therapy response comprises inferring therapy success for therapy applied to the tissue region, and/or wherein classifying the therapy response comprises identifying an example treatment and outcome for another patient with similar extracted values of the features, and transmitting the therapy response. receiving scanning data of a patient scanned with a medical scanner, the scanning data providing multiple frames of data representing a tissue region of interest in the patient; extracting, by a processor, values for features from the frames of data; classifying, by a machine-learnt classifier implemented by the processor, a therapy response of the tissue of the tissue region from the values of the features as input to the machine-learnt classifier; wherein extracting and classifying comprise extracting and classifying based on Siamese convolutional networks comprising two parallel networks which process pre and post treatment data, wherein the Siamese convolutional networks are trained to extract features from given multi-modal, multi-dimensional images, IM1 and IM2, which are used as input images, wherein one branch of the Siamese convolutional network is used to input IM1 representing pre-treatment and another branch is used to input IM2 representing post-treatment, wherein relevant features are automatically determined as part of training, wherein the two parallel networks are trained with exactly the same parameters, and the Siamese convolutional networks are linked so that the same weights for the parameters defining the networks are used in both branches, wherein classifying the therapy response comprises inferring therapy success for therapy applied to the tissue region, and/or wherein classifying the therapy response comprises identifying an example treatment and outcome for another patient with similar extracted values of the features, and wherein extracting and classifying comprise extracting and classifying based on Siamese convolutional networks comprising two parallel networks which process pre and post treatment data, wherein the Siamese convolutional networks are trained to extract features from given multi-modal, multi-dimensional images, IM1 and IM2, which are used as input images, wherein one branch of the Siamese convolutional network is used to input IM1 representing pre-treatment and another branch is used to input IM2 representing post-treatment, wherein relevant features are automatically determined as part of training, wherein the two parallel networks are trained with exactly the same parameters, and the Siamese convolutional networks are linked so that the same weights for the parameters defining the networks are used in both branches, wherein classifying the therapy response comprises inferring therapy success for therapy applied to the tissue region, and/or wherein classifying the therapy response comprises identifying an example treatment and outcome for another patient with similar extracted values of the features, and transmitting the therapy response.