BoneMet: An Open Large-Scale Multi-Modal Murine Dataset for Breast Cancer Bone Metastasis Diagnosis and Prognosis
Authors: Tiankuo Chu
Fudong Lin
Shubo Wang
Jason Jiang
Wiley Jia-Wei Gong
Xu Yuan
Liyun Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Medical Dataset
Breast Cancer Bone Metastasis
Diagnosis
Prognosis
Sparse CT reconstruction
CT
X-ray
Large language model
AI for Science
Primary area: datasets and benchmarks

Abstract: Breast cancer bone metastasis (BCBM) affects women’s health globally, calling
 for the development of effective diagnosis and prognosis solutions. While deep
 learning has exhibited impressive capacities across various healthcare domains, its
 applicability in BCBM diseases is consistently hindered by the lack of an open,
 large-scale, deep learning-ready dataset. As such, we introduce the Bone Metastasis
 (BoneMet) dataset, the first large-scale, publicly available, high-resolution medical
 resource, which is derived from a well-accepted murine BCBM model. The unique
 advantage of BoneMet over existing human datasets is repeated sequential scans
 per subject over the entire disease development phases. The dataset consists of
 over 67 terabytes of multi-modal medical data, including 2D X-ray images, 3D
 CT scans, and detailed biological data (e.g., medical records and bone quantitative
 analysis), collected from more than five hundreds mice spanning from 2019 to
 2024. Our BoneMet dataset is well-organized into six components, i.e., Rotation
X-Ray, Recon-CT, Seg-CT, Regist-CT, RoI-CT, and MiceMediRec. We further
 show that BoneMet can be readily adopted to build versatile, large-scale AI models
 for managing BCBM diseases in terms of diagnosis using 2D or 3D images, prognosis of bone deterioration, and sparse-angle 3D reconstruction for safe long-term
 disease monitoring. Our preliminary results demonstrate that BoneMet has the
 potentials to jump-start the development and fine-tuning of AI-driven solutions
 prior to their applications to human patients. To facilitate its easy access and
 wide dissemination, we have created the BoneMet package, providing three APIs
 that enable researchers to (i) flexibly process and download the BoneMet data
 filtered by specific time frames; and (ii) develop and train large-scale AI models for
 precise BCBM diagnosis and prognosis. The BoneMet dataset is officially available on Hugging Face Datasets at https://huggingface.co/datasets/BoneMet/BoneMet. The BoneMet package is available on the Python Package Index (PyPI) at https://pypi.org/project/BoneMet. Code and tutorials are available at https://github.com/Tiankuo528/BoneMet.

Published as a conference paper at ICLR 2025
BONEMET: AN OPEN LARGE-SCALE MULTI-MODAL MURINE DATASET FOR BREAST CANCER BONE METAS- TASIS DIAGNOSIS AND PROGNOSIS
Tiankuo Chu', Fudong Lin", Shubo Wang', Jason Jiang', Wiley J.W. Gong', Xu Yuan Liyun Wang!
1 "Department of Mechanical Engineering, University of Delaware 2 Department of Computer & Information Sciences, University of Delaware 3 "Department of Computer Science, University of Maryland
ABSTRACT
Breast cancer bone metastasis (BCBM) affects women's health globally, calling for the development of effective diagnosis and prognosis solutions. While deep learning has exhibited impressive capacities across various healthcare domains, its upplicability in BCBM diseases is consistently hindered by the lack of an open, large-scale, deep learning-ready dataset. As such, introduce the Bone Metastasis (BoneMet) dataset, the first large-scale, publicly available, high-resolution medical resource, which is derived from well-accepted murine BCBM model. The unique advantage of BoneMet over existing human datasets is repeated sequential scans per subject over the entire disease development phases. The dataset consists of over 67 terabytes of multi-modal medical data, including 2D X-ray images, 3D CT scans, and detailed biological data (e.g., medical records and bone quantitative analysis), collected from more than five hundreds mice spanning from 2019 to 2024. Our BoneMet dataset is well-organized into six components, i.e., Rotation- X-Ray. Recon-CT, Seg-CT, Regist-CT, RoI-CT, and MiceMediRec. We further show that BoneMet can be readily adopted to build versatile, large-scale Al models for managing BCBM diseases in terms of diagnosis using 2D or 3D images, prog- nosis of bone deterioration, and sparse-angle 3D reconstruction for safe long-term disease monitoring. Our preliminary results demonstrate that BoneMet has the potentials to jump-start the development and fine-tuning of Al-driven solutions prior to their applications
human patients. To facilitate its easy access and wide dissemination, we have created the BoneMet package, providing three APIs that enable researchers to (i) flexibly process and download the BoneMet data filtered by specific time frames; and (ii) develop and train large-scale Al models for precise BCBM diagnosis and prognosis. The BoneMet dataset officially avail- able on Hugging Face Datasets https: /ww.ggigggaac..eecc//////a//aaee co/ /dataassssss s BoneMet BoneMett. The BoneMet package available on the Python Pack- age Index (PyPI) at https: //tp:i ortp Botteeeeet.........tteeee....... Code and tutorials are available at https: http://:////.tttt..c.o..oa//////
BoneMMet.
INTRODUCTION
Breast cancer stands as one of the most prevalent non-skin cancers affecting women globally (6). Although the survival rate for breast cancer patients has been greatly improved due to the adoption of cancer screening, early diagnosis, and more effective cancer treatments, the metastasis of breast cancer other organs including bone dramatically reduces the five-year survival rate and worsens the patient suffering including severe pain, impaired mobility, and elevated risk of fatal fractures (11). Once invading the bone, breast cancers activate bone resorption, which turn drives cancer growth. The "vicious cycle" between tumor and bone leads to the rapid decline of bone structural integrity (12), The
"Coneesooning author: Dr. Xu Yuan (xyuan@ (xyuan@udel.edu)) 'Coresponding author: Dr. Liyun Wang (lywang (ude.eedd

---

Published as a conference paper at ICLR 2025
conventional approaches for managing Breast cancer bone metastasis (BCBM) diseases utilize various medical imaging modalities including i) 2D X-ray images (13; 30), ii) 3D computed tomography (CT) scans (32; 3), and iii) combined localized metabolic imaging and CT scans (8; 46). However, these imaging based diagnosis and prognosis approaches suffer from several limitations, such as low sensitivity small lesions in early-stage bone metastases (10), health risks of CT due to the exposure to high ionizing radiation (5), and interpretation variability limited by radiologist's 's experience and expertise (33). The lack of sensitivity and specificity of the imaging tools and algorithms leads the current clinical guidelines against active monitoring of BCBM because the harms of false positive diagnosis and radiation from longitudinal imaging outweigh potential benefits (15). This poses a
classic catch-22 dilemma. With clinicians being hesitant to image, we cannot feed and train Al models for better diagnosis. Conversely, without effective early detection tools, we miss the critical window of treating BCBM and fail to improve patient outcomes in meaningful way.
Deep learning-based methods have shown significant improvements in identifying subtle patterns and features that are likely missed by human eyes, in particularly when analyzing large volumes of longitudinal data, reducing the scanning duration (radiation exposure) with efficient 3D CT reconstruction, and providing consistent and objective analysis and more comprehensive insights into patient health (24; 23; 44; 36). For BCBM diagnosis and prognosis applications, previous studies (29; 47; 18) have attempted to leverage deep learning models, while their performances appear to be hindered with low resolution images, inadequate diagnosis/prognnsiis labels (17), limited dataset sizes (1; 48; 14), and lack of diverse modalities (20). Hence, although the utilization of AI models to manage BCBM diseases offers a promising direction, the scarcity of large-scale, high- quality, relevant
high-resolution X-ray and CT images along with detailed medical records -ecatlsgreeatly impedes the development and application of large-scale Al models.
To further unleash the power of deep learning in BCBM diagnosis and prognosis, we first attack the major obstacle- the scarcity of high-resolution image datasets of early-stage BCBM with sufficient diagnostic and prognostic labeling- with our open-sourced large-scale BoneMet datasets. We leverage the large preclinical images acquired over the past five years (i.e., 2019-2024) from well- established mouse BCBM models (39). Mouse models, despite limitations, are valuable tools in elucidating disease mechanisms, identifying diagnostic biomarkers, and testing treatments (9). The small-sized mouse skeleton allows quick three- dimee-iimeiionall CT scans (< 4 min per scan) at high resolution (7-10 micron per pixel) and low radiation exposure (<0.4 Gy) for multiple time points over the span of breast cancer metastasis (39). Additional benefits include i) the complete contro//docuuentttiio of the age of the experimental mice, the bone site of cancer cell metastasis, and the type and number of breast cancer cells being introduced in each mouse, ii) the application of various treatments mimic human patient conditions such as chemo and radiation therapy, as well as exercise regimen (40). The rigorous experimental design contains age-matched non-tumor and placebo-treate controls, and thus the acquired image dataset reflects the complexity of human breast cancer progression with superior image quality and labeling over available patient data. We further demonstrate the utility of our BoneMet datasets in supporting the development of imaging biomarkers and AI applications, such as multimodal large vision models (MLVM) with temporal and spatial alignment, for the early diagnosis and prognosis breast cancer bone metastasis as well as improved predictive accuracy of the models used in clinical settings.
Specifically, our dataset is collected from more than five hundreds mice spanning 5 years from 2019 to 2024, with its total size over 67 terabytes. Each mouse undergoes 4 to 5 weekly sequential skeletal scans, including scans of the tibiae, distal femurs and some vertebrae. Each session produces 260 high-resolution, multi-angle 2D X-ray images at 0.8 intervals. From these X-ray images, 3D reconstructed CT images, segmented and registered CT scans, and Rol-cropped CT images of the tibiae are obtained using processing tools such as commercial software and our developed APIs. In addition, detailed biological data is recorded for each mouse, capturing critical medical details such as age, body weight, sex, metastatic tumor growth status, bone structural and mechanical properties, and other relevant data at each time point. To facilitate its effective use, this dataset is well-organized Reconstructed CT Imagery
into six key components: Rotational X-Ray Imagery (Rotation-X-Ray (Recon-CT), Segmented CT Imagery (Seg-CT), Registered CT Imagery CT), Region of Interest CT Imagery (Rol-CT), and Mice Medical Records (MiceMediRee)) Figure provides an overview of our BoneMet dataset, showcasing examples of its six components, the processing tools used to acquire them, and the detailed data collection procedures. Besides, we have developed the BoneMet package, which includes three types of APIs for CT image segmentation, CT image
2

---

Published as a conference paper at ICLR 2025
Rotation-X
In vivo CT scans
SAy
Motaiing waneing
Sep-API
BoneMet
Segreeniation
Dotasloeer
(Prodig Pess-ty)
Data Collection Processing T Tool Six Sub Datasets
Mice edittoc
3. Sege-CT
Dataset
Hespttration
Quaniliotts analyyl
CLASS, Alheguo,,
R5A-G25588d Anaboor
Bel Croyplag Image ancelaps
Regist-APP
Sinpaance suazs Regist-CT
Rol-API CTAse
Rol-CT
Figure I: The data collection, processing tools, and six components for our BoneMet dataset.
registration, and Rol-based CT image cropping, respectively. These tools have been released on the Python Package Index (PyPI), aimed at assisting researchers and practitioners by providing efficient, standardized methods for data processing. These APIs enhance reproducibility, streamline the processing of large datasets, offer user-friendly interfaces, and ensure consistent data handling.
To the best our knowledge, this dataset the first of its kind, offering large-scale, high-resolution, and multi-modal medical data specifically targeting the management of BCBM diseases. It can support both supervised BCBM diagnosis and prognosis, thanks to its ground-truth labels, and self-supervised pre-training, due to its vast amount of data samples. Our experimental results validate the comprehensive applicability our BoneMet dataset.
2 OUR BONEMET DATASET
Our BoneMet dataset is open, large-scale, and multi-modal resource specifically designed for BCBM diseases. offers high-resolution multi-modal medical data, including 2D X-ray images, 3D CT scans, and medical records and quantitative analysis. This section details the collection and preparation process of BoneMet dataset, its contents, and potential applications.
2.1 DATA COLLECTION AND PREPARATION
This dataset spans five years, from 2019 2024, and includes data over 500 either C57BL/6J or Balb/C mice with the age varying from young (8 weeks) to aged (>70 weeks) mice. These mice were divided randomly assigned into non-tumor and tumor groups, which further received individual or combined cancer interventions such as exercise, radiation therapy, chemotherapy and other experimental drugs (39). Breast cancer cells (1000-20,000 Py81 cells or 3000-5000 4TI cells) were directly injected into the proximal end of the mouse tibia of the tumor groups. Some mice received tumor injections on both tibiae, while some mice received injection on one tibia. The tumor and non-tumor carrying mice undergoing interventions placebo treatments were then longitudinally monitored by microCT scans over a period of 3-5 weeks. The overview of data collection and preparation procedures is illustrated in Figure 1, and the details are presented as below, which produce the BoneMet dataset into six well-organized components.
Fürst, the Rotational X-Ray Imagery (Rotation---Ray is captured weekly by the SkyScan@ 1276 scanner (Bruker, Kontich, Belgium), starting from the beginning of the breast tumor inoculation to the end of the sacrifice of the mice for 3 to 5 weeks. The camera revolved around the anesthetized animal's 's body from the anterior (front view) to the lateral and posterior (back view) aspects of the hind limbs with step of 0.8 degrees, with 260 X-ray projections for each scan and covering total rotation of 2 The Rotation-X-Pe X-Ray dataset collected in each week are organized for temporal alignment. The total scanning process for each mouse takes around 10 minutes.
3

---

Published as a conference paper at ICLR 2025
Second, these 260 X-ray images are taken from variable angles reconstructed by NRecon& with voxel size of to 10.6 um as the Reconstructed CT Imagery (Recon-CT). The 3D reconstructed CT is acquired based on the traditional filtered backprojection (FBP), which uses a ID filter on the projection data before backprojecting (2D or 3D) the data onto the image space. Then, the 3D bone reconstructed CT is optimized to get the high quality reconstructed CT images. The reconstruction process takes over 30 minutes per mouse.
Third, among the Recon-CT imagery, both left and right tibiae are segmented from the knee by either our Seg-APl or Dataviewer@) software as the Segmented CT Imagery (Seg-CT). A global threshold value (75/255) applied and the processed images were found to agree well with gray-scaled images. The segmentation of both left and right tibiae takes around 10 to 15 minutes per mouse.
Fourth, after segmentation, each tibia is registered to their reference (vertically aligned tibia) by our Regist-APl or Dataviewer@ software as the Registered CT Imagery (Regist-CT), For the first scan (week 0) of the time sequence, each segmented tibia is aligned to a reference tibia with its long axis vertical and its anterior-pe anterior-posterior and medial-lateral axes orthogonally positioned. Subsequent scans of the segmented tibia captured at later time points are aligned to their baseline (week 0). The registration process of each tibia to their references takes about 15 minutes.
Fifth, the region of interest of tibiae (proximal end where the metastatic tumors are located) overlaid to their baseline is selected by our RoI-API and the fibular are cropped manually by CTAn@ as the Region of Interest CT Imagery (RoI-CT). The overlapped composite Rol-CT images are generated with different pixel values assigned in each pixel based various criterion: the pixel with bone (above threshold value of 75) both in baseline and sequential CTs are assigned to the value of 180 (light gray), the non-bone pixel (below threshold) week 0 later became bone (above threshold) is assigned to 240 (white), the bone pixels (above threshold) at week 0 later became non-bone (below threshold) is set to 60 (dark gray), the pixel without bone in both time points were given the value to 0 (black). After the generation of CT composite, the Rol section is selected with the proximal tibia-fibula junction as the landmark, and the fibula was manually cropped by CTAn@, This process takes approximately 20 minutes for each registered tibia.
Sixth, the quantitative analysis of the RoI-CT imagery by CTAn@, Abaqus@, and RSA-G2 Solids Analyzer@, and the Pre-Op (Pre- Pre-Oppeation) and Post-Op (Post-Operation) medical records of the mice with bone metastasis are combined as the Mice Medical Records (MiceMediRec). The imaging diagnosis parameters include records of tumor growth status (if available), bone structural properties, and mechanical properties. Tumor burden and growth are quantified via IVIS imaging of lumineseence-tageed tumor cells, with the average radiant efficiency measured from the tibia reported. Bone structural properties, such as bone lesion occurrence, total bone volume changes, cortical bone parameters (resorption, area, and thickness), and trabecular bone parameters (bone volume fraction, number, thickness, and spacing). as well as bone mechanical properties (polar moment of inertia (pMOI)) are measured by CTAn@ based on the CT in the Rol-CT dataset. For some samples, additional mechanical properties such as bone stiffness, yield strength, ultimate strength, and work to fracture are simulated or measured by finite element analysis or 3-point bending tests. This process takes over 30 minutes for each mouse.
Due to the page limit, further details on data collection and preparation processes, including breast tumor tibial inoculation and medical image processing, are provided in Section A of the supplementary materials.
2.2 DETAILS OF BONEMET DATASET
The total size of our BoneMet dataset is 67.87 TB, which includes six components, detailed as below. The former five components are all visual data, stored in PNG format, while the last one is numerical data, stored in CSV format.
Rotation-X-Ray The Rotational X-Ray Imagery consists of 651,300 X-ray images of subjects with tumors and 676,000 X-ray images of subjects without tumors. Each image has a resolution of .032x4.032xx pixels and spatial resolution of 0.8", captured at the hindlimb. This dataset has been aligned both spatially and temporally with the temporal resolution of I week, and it offers 2D X-ray images taken from multiple angles, from anterior (front) to lateral (side) and posterior (back) views, providing a comprehensive examination of the subject. The total size of this imagery is 20.93 TB. The left part of Figure 2 shows examples of the 2D X-ray images.
4

---

Published as a conference paper at ICLR 2025
Recon-CT. The Recon-CT dataset comprises 3D CT scans reconstructed from Rotation X-Ray imagery and therefore also temporally aligned with the temporal resolution of week. The reconstruction process is illustrated in the right part of Figure 2. These slices capture cross cross-sectional views of the tibia, femur, and spine, with the dimensions varying according to the specific regions of interest (Rol) identified during the micro-CT reconstruction process. This component includes 2,505 CT scans of subjects with tumors and 2,600 CT scans of subjects without tumors. Each CT scan is composed of 2,685 2D slices with an image resolution of approximately 2,588x2,428x pixels, with an example shown in the left image of Figure 3. The total size of this dataset is 45.23 TB.
Reconimaation
20P robation
per X-0X-r projections
Figure 2: Examples of our Rotation- X- X-Ray and Recon-CT imagery. Left: 2D X-ray images are taken from 260 variable angles with 0.8 intervals of the hindlimb of mouse; and Right: The paired 3D CT scan is reconstructed from these X-ray images.
Seg-CT. These 3D CTs of tibiae are isolated from the 3D CT scans of hindlimb in the Recon-CT imagery, as illustrated on the right side of Figure 3. This component includes 3,005 segmented CT scans of subjects with tumors and 7,205 segmented CT scans of subjects without tumors. Each scan is composed of approximately 1,700#200 2D slices with an image resolution of approximately 700+50x900+8801 pixels. The size of this dataset 1.53 TB.
Recoes
Figure 3: Examples 3D CT scans from the Recon-CT and the Seg-CT imagery. Left: 3D CT scans of hindlimbs in the Recon-CT imagery; and Right: 3D CT scans of segmented tibiae in the Seg-CT imagery. Notably, 3D CT scans are composed of 2D cross-sectional slices.
Regist-CT. This imagery includes registered 3D CT scans tibiae taken various time points and from different animals, aligned to a reference, as shown in Figure at Section A of supplementary materials. This component includes 3,005 registered CT scans of subjects with tumors and 7,205 registered CT scans of subjects without tumors. Each scan is composed of 1,538 2D slices with an image resolution of 509x539x1 pixels. The size of this dataset 0.18 TB.
This imagery focuses on the proximal end sections of the registered tibiae, where the effects of metastasis are most pronounced, as shown in Figure 4. The Rol-CT imagery comprises 300 slices below the proximal tibia-fibula junction, with overlaid registered CT scans aligned to their baseline (week 0). each 2D slice, light gray represents the reserved bone in the sequential scans, white indicates bone formation where non-bone pixels at week 0 later became bone, and dark gray signifies bone resorption where bone pixels at week O later became non-bone. This component includes 3,005 CT scans of the proximal end sections of registered tibiac with tumors and 7, 7,205 CT scans that without tumors. Each 2D slice has the image resolution of 509x539x1 pixels. The size of this dataset is 8.00 GB.
RoI-CT.
5

---

Published as a conference paper at ICLR 2025
Normal
Bone metastatlc tumor
Figure 4: Examples of 3D CT scans from the RoI-CT imagery, including the proximal end of tibiae without (Left) and with (Right) metastatic breast tumors.
MiceMediRec. The Mice Medical Records includes detailed records for individual mice regarding animal ID, strain, date of birth, experiment data, tumor inoculation (left, right, or both tibiac), treatment type, treatment regimen (dose, frequency or duration), scan time and frequency. If available, quantitative analyses of bone from scans, FE simulations, IVIS readouts, and mechanical testing are also included to provide a comprehensive overview of the animals, their bones, and their disease conditions. More details can found Supplementary Materials, Section A, Table The size of this dataset is 9.44 MB.
Table lists more detailed and comprchensive information for our BoneMet dataset and its six components.
Table I: Overview of our BoneMet dataset details.
Neetter of BeageoRoccet Negaise comis are tumer rnce asigned on ne anmaa lovtl
Datacco
Daa Foznz
Stc
Auenerical Rajine
Theeporal Roclltion
Canani
Xenn isago lves in barni tuakk vice!
Negatoe: 676
Stacked
Biomerr
Prodin:: 2,566
CT
2,585 (280))
Hindlezb
wode
inataaly uupponall
chete sectura ot CTT
with
Hecaticn N--lay
CT
Negatin:: 7,266
(Euch wibh 1,00 (200) alica)
900 12500
Tikke
lose Rocee-CT
regrunzed
rngi
ResbeCT
82
Preatre: 1145 Negatie: 7,38
(lach with 1,318 alco)
3D CT
500 $39
Tittac
I wock
ditand dlggned
ventially aigned
RekeT
Pootive: Negatie
3D CT
steed)
Poscisal and
The wilcadd I'l noer oaiic inamea
Medied receal and of 3D CT
MixMMedRRcc
1-4
Negatta $30
Pwittwe: 501
Taxt
NA
saggs dinddaiio and
2.3 POTENTIAL APPLICATIONS
There many applications be supported by BoneMet database. Due to page limits, we only validate some critical applications in Section 3, including i) 2D spatial-temporal X-ray imagery-based BCBM diagnosis; ii) 3D CT imagery-based BCBM diagnosis; and iii) 3D multi-modal Rol-CT imagery-based BCBM prognosis; and iv) Sparse-angle 3D CT reconstruction with one real 2D X-ray images. Besides, more supportive applications, such 3D CT auto-segmentation, generalist biomedical AI diagnosis, among others, are listed in Supplementary Materials, Section A, Table 6 to exhibit its broader applicability.
EXPERIMENTS AND RESULTS
We conduct experiments exhibit its applicability and efficiency to manage BCBM disease. Currently, the positive and negative labels assigned at the animal level, rather than individual time points and individual X-ray images. positive label of a mouse indicates that a metastatic bone lesion occurs in the subject between week o and week 5. There are animal-to-animal variations in the times of bone lesion initiated and the speed of lesion growth.
BoneMet dataset via developing various deep learning solutions to
3.1 HYPERPARAMETERS AND DATASET SPLITTING
The hyperparameters used in our Breast cancer bone metastasis (BCBM) diagnosis experiments follow the common practice of supervised ViT training. The key hyperparameters are detailed in Table 2.

---

Published as a conference paper at ICLR 2025
For the experiments presented in our manuscript, we employ train/test split with ratio of 8:2. Specifically, 80% of the mice were used for training the model, and the remaining 20% were reserved for testing. Within the training dataset, we implemented 5-fold cross-validation strategy.
This process is repeated five times, with each subset serv-
ing the validation set once, so the validation data is never seen by the model during the training phases, ensuring that there is no data leakage. In our trials, the data split is con-
Table 2: Training Configuration
Comfii
Value
Optimizze
Adam' le:3
ducted at the mice level, so the entire sets of images from
Bise leamiing rate
Weight decay
0.05
individual mice were allocated either to the training or test-
Optanizer
P1. Cosine docay
ing datasets. This is to ensure no overlap of data from the same mice between the training and testing datasets, pre- venting any potential data leakage. All our experiments are conducted on a high-performn perfoorrformence workstation equipped with an NVIDIA RTX A6000 GPU, which has 48GB of VRAM.
Learning rase schedale
Warsup spochs
5
Training opechs
100
Augmentation
RandAng (9. 0.5)
3.2 APPLICABILITY OF OUR ROTATION-X-RA DATASET FOR BCBM DIAGNOSIS
We conduct experiments to demonstrate the applicability of our Rotation-X X-Ray dataset to manage BCBM disease. Here, we employ two Vision Transformers (ViT) variants for the diagnosis of BCBM using Rotation-X-RP --aay imagery: a simple VIT variant Swin-Base (25), and spatial-temporal VIT variant MMST-VIT 1).. Notably, the 2D X-ray images within the Rotation-X- --Ray imagery have been carefully spatially and temporally aligned. For clarity, we denote MIST-ViT as VIT (w/ STA), indicating its ability to leverage spatial-temporal alignment (STA), and Swin-Base, as ViT (w/o STA), indicating not utilizing STA. Table 3 presents the performance outcomes measured by the metrics of Precision, Recall, F1-Score, and Accuracy. We have two observations. First, both ViT (w/o STA) and ViT (w/ STA) can achieve decent diagnosis performance results, with the overall accuracies of 79.19 1% and 89.0%, respectively. This suggests that our dataset is adaptable to various ViT variants, from simple to more complex architectures, for effective BCBM diagnosis. Second, we observe significant training-test accuracy gap of 20.4% with VIT (w/o STA), indicating pronounced overfitting issue inherent to the ViT architecture. In contrast, the incorporation STA in the Rotation-X-Ba X-Ray imagery substantially alleviates this issue, as demonstrated by the notably smaller teeting-tttttee test accuracy gap of 6.0% achieved by ViT (w/ STA). This highlights the effectiveness of STA in mitigating overfitting and enhancing model generalizability our Rotation X- X-Ray dataset.
training-
Table 3: The BCBM diagnosis using VIT with and without STA on the Rotation- Maatiion---aayy X-Ray imagery
Training
Tost
Methods
Precision
Recall
FI-Score
Accuracy
Precision
Recall
F1-Score
Accuracy
VIT (wl STA)
99.6 95.4
99.7
99.7 96
99.5
92.3
80.5 90.6
86.0
79.1
VIT (w/ STA)
96.9
95.0
92.1
91.3
89.0
3.3 APPLICABILITY OF 3D CT SCANS FOR BCBM DIAGNOSIS ACROSS VARIOUS MODEL ARCHITECTURES
Here, we conduct experiments by utilizing 3D CT scans in the Regist-CT imagery for BCBM diagno- sis. We utilize two CNN-based model M (35)-as along with one ViT-based model architecture, i.e., Swin-B, as the backbone networks. These models extract slice-level features from each 2D slice within the CT volume and subsequently aggregate these features into volume-level representations using max-pooling layer.
BBggttrree--BBTasssseeeee (BiT-M) (16) and EfficientNert2-
Figures 5a, 5b, 5c, and 5d present the performance results. We observe BiT, EfficientNerV2, and Swin-B exhibit commendable performance, achieving overall accuracies of 81. 81.0%, 85.0%, and 92.0% respectively. These results validate that our BoneMet dataset is compatible with both -bNed and ViT-based model architectures, demonstrating its broad applicability. Furthermore, the Swin model consistently outperforms its two CNN- counterparts. For instance, in terms of the FI-Score. it exceeds BiT and EfficientNetV2 by 9.3% and 5.9%, respectively. This superior performance is attributed to the ViT-based model's use of Multi-head Self-Attention (MSA) (38), which enhances its ability to effectively aggregate slice-level features into volume-level representations.
7

---

Published as a conference paper at ICLR 2025
(a) Precision
(b) Recall
(c) F1-Score
(d) Accuracy
Figure 5: The diagnosis of BCBM using 3D CT scans from the Regist-CT imagery.
3.4 APPLICABILITY OF Rol-CT IMAGERY AND MICEMEDIREC DATASETS FOR MULTI-MODAL PROGNOSTIC ASSESSMENT BONE MECHANICAL PROPERTIES
Managing pathological fractures associated with bone metastases is crucial for preserving patient's mobility and quality of life. In this context, we explore the applicability of our Rol-CT dataset to this critical scenario by assessing the mechanical competence of the bone future time point. We formulate this task as multi-modal prognostic assessment of bone mechanical properties. First, the 3D CT scans in the Rol-CT imagery are used train generative models to produce 3D CT scans of future frames, reflecting the progression of bone lesions. Next, the biological data in the MiceMediRec component is utilized to simulate the mechanical behaviors of axial compression of the proximal end of tibiae with metastatic osteolysis.
Three generative models are taken into account: 3D Generative Adversarial Networks (3D-GAN) (2), Temporal Variational Autoencoders (T-VAE) (43), and (ST-VAE), for generating future 3D CT scans. The quality of these generated scans is measured using PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity). and I.PIS (I (Learned Perceptual Image Patch Similarity) metrics. Note that a higher PSNR (and SSIM) value or a lower L.PIS value indicates better generation quality. Table 4 presents the quantitative results. We observe that all three methods achieve high-quality future CT generations. For instance, the 3D-GAN, T-VAE, and VAE methods achieve SSIM values of 0.767, 0.817, and 0.860, respectively.
Variational Autoencoders
Table 4: Evaluations of the quality of 3D CT prediction using three metrics
Methods
PSNR
SSIM
LPIPS (J)
3D-GAN
21.9
0.767
T-VAE
23.4
0.817
0.078
ST-VAE
35.5
0.860
0.041
Force va Displocement
CT G Geseration
FE simulation
Gread
Dipiiccmeri.cco (MK)
(a) CT generation and FE simulation
(b) Reaction force prediction
Figure 6: Evaluations of prognostic assessment of bone mechanical properties: (a) comparisons 3D CT scan and its ground truth, along with the corresponding finite element (FE) analysis of axial compression of proximal tibiae; and (b) comparisons between the predicted and ground truth reaction force values various displacement levels.
between ST-VAE-genernee
The visualization and quantitative results obtained using the ST-VAE model shows the quality of 3D CT prognosis, as exhibited in Figure 6. In Figure 6a, the predicted 3D rendering volume is comparable to the ground truth, and the finite element (FE) simulations of axial compression on meshes constructed from the predicted bone structures closely resemble those constructed from the ground truth. Furthermore, Figure 6b illustrates the curves of the ground truth and predicted reaction force values various displacement levels. It evident that the predicted reaction force curve closely
8

---

Published as a conference paper at ICLR 2025
matches the ground truth, achieving impressive R² value of 0.956. These results underscore the significance our BoneMet data in the multi multi-modal prognostic assessment of bone mechanical properties with metastatic bone lesions.
3.5 APPLICABILITY OF SPARSE-ANGLE CT RECONSTRUCTION
Conventional CT reconstruction methods require 2D X-ray images from multiple angles. For example, using NRecon@ software necessitates at least 260 X-ray images from different angles for accurate reconstruction. This leads to prolonged exposure to ionizing radiation during the process capturing multiple 2D X-ray images, incurring severe adverse effects the patient's health. Here, explore the applicability of our Rotation-X-Ree dataset to sparse-angle CT reconstruction, which can be validated by paired CTs in Recon-CT dataset. This task can significantly reduce radiation exposure by reconstructing 3D CT scans with continuous viewpoint rotations from a single 2D X-ray image.
We have employed two NeRF-based methods: PixelNeRF (45) and MedNeRF (4), reconstruct 3D CT scans. The reconstruction quality is measured by the PSNR (Peak Signal-to-Noiss Ratio), SSIM (Structural Similarity Index), FID (Fréchet Inception Distance), and KID (Kernel Inception Distance) metrics. Notably, higher PSNR and SSIM values, along with lower FID and KID values, indicate better reconstruction quality. Table presents the quantitative results of reconstructing 3D CT scans. It is observed that our dataset can support both NeRF-based methods for sparse-angle reconstruction. For example, the MedNeRF can achieve superb reconstruction quality, with high PSNR (and SSIM) value of 30.2 (and 0.810) and a low FID (and KID) value of 91.5 (and 0.092). Moreover, we use the MedNeRF method to generate a complete set CT projections within full vertical rotation from given single-view X-ray of a CT slice, with results shown in Figure 7. Despite the challenge of this task, the MedNeRF method produces high-quality reconstructions, demonstrating the applicabiliiy of our BoneMet dataset for sparse-angle CT reconstruction.
Table 5: Quantitative evaluations of sparse- angle CT reconstruction
Methods
PSNR (f)
SSIM (t)
FID (1)
KID(L)
PixeNNeR MedNeRF
155.2
0.128
30.2
0.810
91.5
0.092
GhmmXny
Gonund
Figure 7: Illustration of sparse-angle CT reconstruction from a X-ray image by MedNeRF method. The first row also presents the given 2D X-ray image and X-ray images at other angles for reference. The second row shows the reconstructed 3D CT scans, while the third row displays the ground-truth 3D CT scans from the Recon-CT imagery.
We have also conducted additional experiments, including tibiae auto-segmentation, to validate the significance and broad applicability of our Recon-CT and Seg-CT dataset, and generalist biomedical Al diagnosis of metastatic breast tumor bone, to demonstrate the applicability of our Rol-CT and MiceMediRee dataset. The details of these experiments are deferred to Sections B of the supplementary materials. Besides, the future potential applications of the BoneMet dataset discussed in Sections D of the supplementary materials.
THE BONEMET PACKAGE
In addition BoneMet dataset, also develop the BoneMet package, including three types of APIs: 1) CT Image Segmentation, 2) CT Image Registration, and 3) Rol-based CT Image Cropping), the Python Package Index( for public release to facilitate dataset's ease access. The details of three APIs and their usage examples are deferred to Section of supplementary materials.

---

Published as a conference paper at ICLR 2025
5 CONCLUSIONS
This paper introduces the BoneMet dataset, the first large-scale meticulously curated collection of well-organized CT and X-ray images in angles, positions and time points designed specifically for the diagnosis and prognosis of BCBM discase. It includes high-resolution images that are well-organized across spatial and temporal dimensions, allowing for an in-depth analysis of bone responses to tumor progression and treatment. The integration of detailed imaging with medical metadata enhances the utility of the BoneMet dataset, enabling the development and validation of advanced deep learning models. This dataset is also invaluable for developing high-quality CT reconstruction techniques from sparse X-ray data and for performing 3D segmentation. Our extensive experimental evaluations confirm that the dataset compatible with various deep learning approaches. Alongside the dataset, we have developed the BoneMet Package to assist researchers and practitioners in automatic Breast cancer bone metastasis medical images processing such as 3D CT images segmentation, registration and region of interest (Rol) selection, well as constructing their own deep learning models. While the primary aim of the BoneMet dataset is to advance deep learning models in disease diagnosis, management and medical image processing techniques, its potential applications extend across medical imaging, oncology, and computational pathology. We believe that our BoneMet dataset will be significant valuable asset the fields of deep learning, medical radiology, orthopedics and oncology, spurring further interdisciplinary research at the intersection field of healthcare and artificial intelligence.
6 BROADER IMPACTS
The ultimate goal of our project is to unleash the power of deep learning in BCBM diagnosis and prognosis. In this study, we have attempted to attack the major obstacle-the the scarcity of high-resolution image datasets of early-stage BCBM with sufficient diagnostic and prognostic labeling- with our open-sourced large-scale BoneMet datasets based on murine models. Utilizing our datasets, we have demonstrated the feasibility of generating multi-angle x-ray images of tibiae and 3D CT reconstructions from sparse-angle x-rays. These generative 3D reconstruction techniques developed using our murine dataset could be refined and applied to human patient x-ray images which are more commonly used for screening. We believe that this regenerative approach will address the radiation exposure concerns and help develop human breast cancer bone metastasis datasets that cover all disease stages, especially the early phases and include sequential and safe imaging of the patients. We anticipate that the human datasets would open the door to build and validate foundation models for human BCBM. The two tasks could potentially form a positive feedback loop: a high-quality dataset drives the performance of AI learning of more accurate and predictive foundation models, which in turn increases the efficacy of the generative 3D reconstruction algorithm, leading to further successful expansion of the dataset.
We wish to engage AI scientists, biomedical investigators, clinicians, and policymakers in developing more effective diagnosis, prognosis, and treatments of cancer bone metastasis. By publishing our BoneMet dataset, we envision that more stakeholders can benefit from full access to the dataset, which could be used to develop early diagnostic tools for breast cancer metastasis with improved accuracy and sensitivity. The dataset and the insights it generates could play a crucial role advancing AI applications, such as large vision models and multimodal approaches with temporal and spatial alignment, for the early diagnosis and prognosis of breast cancer bone metastasis, particularly in enhancing the predictive accuracy of models used in clinical settings when the disease progression is relatively slower. As we continue to refine and expand our dataset, we aim to provide robust resource for better understanding, detection, and treatment of bone metastases in human breast cancer, and further support future innovations in deep learning studies and clinical practices.
7 ACKNOWLEDGMENTS
This work was partially funded by the following grants: Delaware Center for Musculoskeletal Research (DCMR, 39760) Research Core Access Award (to L. Wang), Women's 's Health supplement (P20GM139760 03W1, to L. Wang), and DCMR pilot project (to X. Yuan). Any opinions and findings expressed in the paper are those of the authors and do not necessarily reflect the view of funding agencies.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
[1] Medpix medical image database. http:://rea.uuuuu....ammmmppappp html, 2024. Free Online Medical Image Database.
[2] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, al. Efficient geometry- aware 3d generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16123-16133, 2022.
[3] Gary JR Cook, Gurdip K Azad, and Vicky Goh. Imaging bone metastases in breast cancer: staging and response assessment. Journal of Nuclear Medicine, 57(Supplement 1):278-33S, 2016.
[4] Abril Corona-Figuerow Jonathan Frawley, Sam Bond-Taylor, Sarath Bethapudi, Hubert PH Shum, and Chris G Willcocks. Mednerf: Medical neural radiance fields for reconstructing 3d-aware ct-projections from single x-ray. In 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pages 3843-3848. IEEE, 2022.
[5] Colleen M Costelloe, Eric M Rohren, John E Madewell, Tsuyoshi Hamaoka, Richard L Theri- ault, Kue-Kuan Yu, Valerae O Lewis, Jingfei Ma, R Jason Stafford, Ana M Tari, al. Imaging bone metastases in breast cancer: techniques and recommendations for diagnosis. The lancet oncology, 10(6):606-6 614, 2009.
[6] Steven Coughlin and Donatus U Ekwueme. Breast cancer as a global health concern. Cancer epidemiology, 33(5):315-3) 2009,
[7] Bob D De Vos, Floris Berendsen, Max A Viergever, Hessam Sokooti, Marius Staring, and Ivana Igumm A deep learning framework for unsupervised affine and deformable image registration. Medical image analysis, 12:128-1433 143, 2019.
[8] Yusuf E Erdi, John I. Humm, Massimo Imbriaco, Henry Yeung, and Steven M Larson. Quanti- tative bone metastases analysis based on image segmentation. Journal of Nuclear Medicine, 18(1):14011146 1997.
[9] Anna Fantozzi and Gerhard Christofori. Mouse models of breast cancer metastasis. Breast Cancer Research, 8:1-11, 2006.
[10] CSB Galasko. Diagnosis of skeletal metastases and assessment of response to treatment. Clinical Orthopaedics and Related Research@, 3 12::4475, 1995.
[11] Margery Gass and Bess Dawson Hughes. Preventing osteoporosis-rclated fractures: an overview. The American journal of medicine, (9(4))S3-S11) 2006.
[12] Theresa A Guise. The vicious cycle of bone metastases. Journal of Musculoskeletal and Neuronal Interactions, 270-577,,, 2002.
[13] Tsuyoshi Hamaoka, John Madewell, Donald Podoloff, Gabriel Hortobagyi, and Naoto Ueno. Bone imaging in metastatic breast cancer. Journal of Clinical Oncology, 22(14):2942- 2953, 2004.
[14] Shih-Cheng Huang, Zepeng Huo, Ethan Steinberg, Chia-Chun Chiang, Matthew P Lungren, Curtis Langlotz, Serena Yeung, Nigam Shah, and Jason Alan Fries. Inspect: A multimodal dataset for pulmonary embolism diagnosis and prognosis. In Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Truck, 2023.
[15] Nancy L Keating, Mary Beth Landrum, Edward Guadagnoli, Eric P Winer, and John Z Ayanian. Surveillance testing among survivors of early-stage breast cancer. Journal of Clinical Oncology. 25(9):1074-1081. 2007.
[16] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European Conference Computer Vision (ECCV), volume 12350, pages 491-507, 2020.

---

Published as a conference paper at ICLR 2025
[17] Ajith AK Kumar. Mortality prediction in the icu: The daunting task of predicting the unpre- dictable. Indian Journal of Critical Care Medicine: Peer-reviewed, Official Publication of Indian Society of Critical Care Medicine, 26(1):13, 2022.
[18] Chaofan Li, Mengjie Liu, Jia Li, Weiwei Wang, Cong Feng, Yifan Cai, Fei Wu, Xixi Zhao, Chong Du, Yinbin Zhang, et al. Machine learning predicts the prognosis of breast cancer patients with initial bone metastases. Frontiers in Public Health, 10:0003,76, 2022.
[19] Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large saguage-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024.
[20] Yikuan Li, Shishir Rao, José Roberto Ayala Solares, Abdelaali Hassaine, Rema Ramakrishnan, Dexter Canoy, Yajie Zhu, Kazem Rahimi, and Gholamreza Kalimi-Khorshian Behrt: transformer for electronic health records. Scientific reports, 10(1):7155, 2020.
[21] Fudong Lin, Summer Crawford, Kaleb Guillot, Yihe Zhang, Yan Chen, Xu Yuan, Li Chen, Shelby Williams, Robert Minvielle, Xiangming Xiao, et al. Mmst-vit: Climate change-aware crop yield prediction via multi-modal spatial-temporal vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5774-5784, 2023.
[22] Lin. Focal loss for dense object detection, arXiv preprint arXiv: / 707.0008..02222 2017.
[23] Jiaming Liu, Rushil Anirudh, Jayaraman Thiagarajan, Stewart He, K Aditya Mohan, Ulugbek Kamilov, and Hyojin Kim. Dolce: model- -baed probabilistic diffusion framework for limited- angle ct reconstruction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 10498-10508, 2023.
[24] Yuan Liu, Ayush Jain, Clara Eng, David H Way, Kang Lee, Peggy Bui, Kimberly Kanada, Guilherme de Oliveira Marinho, Jessica Gallegos, Sara Gabriele, et al. A deep learning system for differential diagnosis of skin diseases. Nature medicine, 26(6):900-908, 2020.
[25] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEESCVF international conference on computer vision, pages 10012-10022, 2021.
[26] Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature Communications, 15(1):654, 2024.
[27] Vir Chau Markus Zhang. Babydoctor. https: http://tthab..ccm/mooot/mm/aaaaa
com
2023. GitHub.
[28] Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature, 616(7956):259225) 2023.
[29] Nikolaos Papandrianos, Elpiniki Papageorgiou, Athanasios Anagnostis, and Anna Feleki. A deep-learning approach for diagnosis of metastatic breast cancer in bones from whole-bode scans. Applied Sciences, 10(3):997, 2020.
[30] DavidJ Perez, Jonathan Milan, HT Ford, JamesS Macdonald, TrevorJ Powles, JC Gazet, V Ralph Mccready, and R Charles Coombes. Detection of breast carcinoma metastases in bone: relative merits of x-rays and skeletal scintigraphy. The Lancet, 322(635)):613-666 1983.
[31] Renzo Phellan, Bahe Hachem, Julien Clin, Jean-Mare Mac-Thiong, and Luc Duong. Real-time biomechanies using the finite element method and machine learning: Review and perspective. Medical Physics, 48(1):7-18, 2021.
[32] Jian Rong, Siyang Wang. Qiue Ding, Miao Yun, Zhousan Zheng, and Sheng Ye. Comparison of 18fdg pet-ct and bone scintigraphy for detection of bone metastases in breast cancer patients. a meta-analysis. Surgical oncology, 22(2):86-91, 2013.
12

---

Published as a conference paper at ICLR 2025
[33] Roman Shyyan, Shahla Masood, Rajendra A Badwe, Kathleen M Errico, Laura Liberman, Vahit Ozmen, Helge Stalsberg, Hernan Vargas, László Vass, Global Summit Diagnosis, and Pathology Panel. Breast cancer in limited-resouree countries: diagnosis and pathology. The breast journal, 12:S27-S37, 2006.
[34] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
[35] Mingxing Tan and Quoc Le. Efficientnetv2: Smaller models and faster training. In International Conference Machine Learning (ICML), pages 10096-101066 2021.
[36] Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 1908):1930-1940, 2023.
[37] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference machine learning, pages 10347-10357. PMLR, 2021.
[38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems (NerulPS), pages
2017.
[39] Shubo Wang, Shaopeng Pei, Murtaza Wasi, Ashutosh Parajuli, Albert Yee, Lidan You, and Liyun Wang. Moderate tibial loading and treadmill running, but not overloading, protect adult murine bone from destruction by metastasized breast cancer. Bone, 153:1 2021.
[40] Murtaza Wasi, Tiankuo Chu, Rosa M Guerra, Rory Kooker, Kenneth Maldonado, Xuehua Li, Chun-Yo Yu Lin, Xin Song, Jinhu Xiong, Lidan You, et al. Mitigating aging and doxorubicin induced bone loss in mature mice via mechanobiology based treatments. Bone, 188: 17235, 2024.
[41] Jakob Wasserthal, Hanns-Christiee Christian Breit, Manfred T Meyer, Maurice Pradella, Daniel Hinck, Alexander W Sauter, Tobias Heye, Daniel T Boll, Joshy Cyriac, Shan Yang, et al. Totalseg- mentator: Robust segmentation of 104 anatomic structures in images. Radiology: Artificial Intelligence, 5(5), 2023.
[42] P Workman, EO Aboagye, F Balkwill, A Balmain, Gail Bruder, DJ Chaplin, JA Double, J Everitt, DAH Farningham, MJ Glennie, et al. Guidelines for the welfare and use of animals in 2010.
cancer research. British journal of cancer, 102((1):1555-
[43] Wei Xiong, Neil Yeung, Shubo Wang, Haofu Liao, Liyun Wang, and Jiebo Luo. Breast cancer induced bone osteolysis prediction using temporal variational autoencoders. BME frontiers, 2022.
[44] Bin Yan and Mingtao Pei. Clinical-bert: Vision- langualangguage pre-training for radiograph diagnosis and reports generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2982-2990, 2022.
[45] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4578-4587, 2021.
[46] Yiqiu Zhang, Beilei Li, Haojun Yu, Junyi Song, Yuhong Zhou, and Hongcheng Shi. The value of skeletal standardized uptake values obtained by quantitative single-photon emission computed -oomppppph- tomography in differential diagnosis of bone metastases. Nuclear Medicine Communications, 42 42(1):63-67, 2021.
[47] Zhen Zhao, Yong Pi, Lisha Jiang, Yongzhao Xiang, Jianan Wei, Pei Yang, Wenjie Zhang, Xiao Zhong, Ke Zhou, Yuhao Li, et al. Deep neural network based artificial intelligence assisted diagnosis of bone scintigraphy for cancer bone metastasis. Scientific reports,
2020.
13

---

Published as a conference paper at ICLR 2025
[48] Zhuo Zhi, Moe Elbadawi, Adam Daneshmend, Mine Orlu, Abdul Basit, Andreas Demosthenous, and Miguel Rodrigues. Multimodal diagnosis for pulmonary embolism from ehr data and ct images. In Engineering in Medicine & Biology Society (EMBC), pages
2022.
[49] Yukun Zhou, Mark Chia, Siegfried K Wagner, Murat S Ayhan, Dominic J Williamson, Robbert R Struyven, Timing Liu, Moucheng Xu, Mateo G Lozano, Peter Woodward-Coute et al. A foundation model for generalizable disease detection from retinal images. Nature, 622(7981):156-163 2023.
OUTLINE
This document provided supplementary materials to support our main paper. Section A provides details of Breast cancer bone metastasis model creations and data collection processes. Section B presents additional experimental settings and results. Section C lists three examples of API for CT Image Segmentation, Registration and Rol-based CT Image Cropping, and section D talks about future potential application of our BoneMet dataset.
A DETAILS OF DATASET AND DATA COLLECTION
In order to build such large-scale BoneMet dataset, the team has spent more than five years with three to four graduate students assisted with over ten undergraduates and summer students. The experimental expenses include mice purchase, shipping, per diem, surgical tools, cell culture, and user fee to access micro-CT scanners, mechanical testers, microscopy. Usually, it takes approximately one year's training to become an expert in inoculating cancer to mice and 2 to 3 month each batch of the mice (around 20 mice) for data acquisition, image processing including CT reconstruction, segmentation, registration, and image analysis.
For each scan, mice were anesthetized with 3% (v/v) isoflurane and held in the built-in holder, which was rotated 208-degree with a step of 0,8 degrees and there are 260 X-ray images taken in total. One frame was taken per step with the following settings: 900 ms exposure time, X-ray of 200 mA current and 50 kVp, and a 0.5 mm Al filter, cone-beam angle Horizontal of 25, 55.888290 (deg) and Vertical 17.423092 (deg). The in vivo scanning of the hindlimbs lasted around min per animal and the accumulated radiation exposure (< 600 mGy) was low.
Image Acquisition:
The 3D volume or 2D cross-sectional images of micro-CT were reconstructed from 260 X-ray projections using the NRecon@ software (Bruker) with voxel size of 7 to 10.6 μm. The 3D reconstructed CT is acquired based on the traditional filtered backprojection (FBP), uses ID filter on the projection data before backprojecting (2D 3D) the data onto the image space. Then, the 3D bone reconstructed CT were optimized with the several steps such as center of rotation adjustment,bean hardening correction,ring artifact reduction and reconstruction filtering in order to get the reconstructed micro-CT images with high quality. Among these steps, center of rotation adjustment used ensure that the reconstruction is symmetrical, beam hardening correction is to compensate artifacts in the scan. Since the 260 X-ray images were taken by the rotation of cameras, the ring artifacts generated during the reconstruction process is minimized by ring artifact reduction. The median filter is employed to remove the salt-and and-pepper noise in the reconstructed images, which the common noise happening in the medical images.
3D Reconstruction:
Segmentation: The the left and right tibiae were segmented from reconstructed CT images by our python segmentation package. A global threshold value (75/255) was applied and the processed images were found to agree well with the gray-scaled images. Two limbs of each mice were separated by their momentum intensity in their relative positions and the tibia and femur of each limb were segmented by identify the specific structure of the knee, where the position of the knee is found by comparing the cross-scctional areas of each 2D slices of the bone and the minimal is where the knee located.
Registration (alignment): After the segmentation, the week O scans of each mice tibia were aligned to registration reference tibia CT, where the long axis positioned vertically and the anterior-postert and medial-laters lateral axes arranged orthogonally, and the transformation and rotation were manually adjust by Dataviwer@. Then the registered tibiae CT week O scans served as baseline scan for
14

---

Published as a conference paper at ICLR 2025
their subsequent week scans. The mutual information maximizing between those of tibiae CT from different mice at week 0 and the registration reference tibia CT, and between those of the sequential scans and themselves week 0 baselines. The mutual information is measure of the statistical dependence or information shared between the image intensities of the two images. Before transformation (rotation, translation, scaling, etc.), initial coarse alignment is performed reduces the search space for the registration algorithm, then transformation applied to the moving image, maximizing the mutual information between the fixed image and the transformed moving image by gradient descent optimization. Once the optimal parameters are found, the final transformation will be applied to the moving image to align it with the fixed image.
Sample
Sample registration
001 left
001 right
002 left
Reference
001 left
001 right
002 left
week
Time regisratioon
week 0
Registration (ignmment
week
week 1
Figure 8: Examples of 3D CT scans from the Seg-CT and the Regist-CT imagery. Left: 3D CT scans from the Seg-CT imagery (i.e, w/o registration); and Right: 3D CT scans from the Regist-CT imagery, which are registered relative to their vertically aligned reference various time points and across different animals.
Selection of Region of Interest: The overlapped composite Rol-CT images were generated with different pixel values assigned in each pixel based on various criterion: the pixel with bone (above threshold value of 75) in baseline and sequential scans were assigned to the value of 180 (light gray), the non-bone pixel (below threshold) week 0 later became bone (above threshold) is assigned to 240 (white),the bone pixels (above threshold) at week 0 later became non-bone (below threshold) is the value to 60 (dark gray), the pixel without bone both time points were gave the value (black). After the generation of CT composites, the Rol section was selected automatically with the proximal tibia-fibula junetion as the landmark, where the number of contoured circles of bone tissues in each 2D slice changes. Then, the fibula in the 2D slices was manually cropped by CTAn@ with interpolated mask.
Analysis and Quantification: The desired analysis such as bone volume changes was conducted on the overlaid tibiae ROI or registered CT to access the bone structure morphology changes, metastatic breast cancer disease progression and mechanical properties impairment over time with and without treatment effects. Cortical polar moment inertia (Ct.pMMI), bone mineral density (Ct.BMD), and tissue mineral density
well trabecular bone volume fraction (Tb.BV/TV), thickness Th), separation (Tb,Sp), bone mineral density (Tb. BMD), and tissue mineral density (Tb.TM) were calculated using CTan@ 3D analysis software. The overlaid subsequent scans with the baseline scan (Week 0) of the same tibia make it possible to quantify the changes (Delta values) of each tibia subsequent weeks relative to week 0.
B SUPPORTING EXPERIMENTAL SETTINGS AND RESULTS
B. TIBIAE AUTO-SEGMENTANG
Conventional nnU-net as well as emerging foundation models such as MedSAM have demonstrated the improved accuracy and robustness for universal medical image segmentation (26). TotalSeg- mentator. which is pretrained on the nnU-Net segmentation algorithm with the dataset 1204 CT examinations, presents deep learning model to automatically and robustly segment all major
15

---

Published as a conference paper at ICLR 2025
Table Overview of the six components in our BoneMet dataset
Components
Advantages
Applications
cost, wide availability, and minimul radiation exposure
Pre-training and fine-tuning deep learning models for bone metastasis eurly detection and momiloring
Rotation-X
Training deep leamning models for hone metastasis localization, segmentation, and sparse angle reconstruction
Reveal the precise location and size bone lesions with detailed 3D structure and 2D cross-soctions
Recon-CT
Training deep learning models for segmentttion and region region-specific analysis of metastatic lesions
Seg-CT
Enhance the diagnosis efliciency by segmenting limited regions
Training deep leurniny models for
Increase sensirivity, making subtle changes bome structures detectable
longitudinal analysis, early dingnosis, prognosis of metastatic changes, diseuse progression
Regist-CT
and treatment effects analysis
Training deep learning models for quamtitative unalysis and prediction of metastatic bone structural mechanical properties
Enable precise diagnosis and quantitative studies
RoI-CT
Detailed demographic information about the animals and the diseuse
Enabling multinodaa deep learning models for comprehensive disease diagnosis, prognosis, and generalist biomedical Al diagnosis
MiceMediRee
Table 7: Details of MiceMediRec Dataset
Source
Parameters
Description
Age
Unit: week
Sex
Medical record
Date
The date cancer inoculation
Body Weight
Unit:
Tibiae cancer inoculation
Records each libise with cancer without cancer Treatment to metastatic breast caneer, like chemotberapy. Dose Unit: μL// Diagnosis bone metastasis, like hone lesion
Treatmeut
Diagnosis
Average Radiant Efficiency
IVIS signal to breast cancer the hone. Unit: pWom?
Bone vulume
Bone total volumes. Unit: mm3
Cort. pM01
Cortical hone polar moment of inertia. Unit: mm3
Quantitative umalysis
Cort. Ar
Cortical bone area. Unit: mm3
Cort. Th
Cortical bone thickness. Unit: mm
Trab BV/TV
Trabecular bone volume relative to the manoo volume. Unit: %
Trab. Th
The average thickness of the individual trabeculac. Unit: mm
Trab, N Trab. Sp
The number of traheculae per unit length. Unit: mm
The average distance betwee trabcculac. Unit: mm
Displacement
the delormation of the bone resporse applied load. Unit:
Reaction force
The force exerted by constraint on the bone in response to an applicd load. Unit: Bone resists deformation in response to applied force. Unit: N/m
Stiffness Vield load
force where hone begins deform permanently. Unit: N
Maximum load Work to fracture
The highest force the bone can safely withstund before failure. Unit: N
The energy bone absorhed hefore fracture. Unit:
anatomic structures on body CT images (41). this experiment, we use the TotalSegmentator to segment the right tibia from hindlimb in our Recon-CT dataset with variable resolutions by down- sizing, and employ our paired Seg-CT dataset as the ground truth. As shown in Figure 9, the left side is the depiction of the 3D render of hindlimb CT from the Recon-CT dataset. After inference, the nnU-Net model successfully identified the left tibia from other part of the hindlimb, including the femur, spine, and hips in all CT's with variable resolutions and most of the tibiae are segmented accurately by the TotalSegmentator, with better segmentation in higher resolution CT when compared with the ground truth tibia from our Seg-CT dataset. This experiment underscores the importance of high resolution dataset for effective bone segmentation. The segmentation of the tibia using the nnU-Net deep learning model hold sigmificant potential for applications in disease characterization. surgical and radiation therapy planning.
B.2 GENERATIVE AI MODEL FOR DIAGNOSIS OF METASTATIC BREAST CANCER ON BONE
Generative AI (GenAl) models can flexibly interpret different combinations of medical modalities, including data from imaging, electronic health records and laboratory results, and in turn produce expressive outputs for disease diagnosis assistance with advanced medical reasoning abilities (28). In this experiment, we used a LLaVA (Large Language and Vision Assistant), which is a combination of the cutting-edge I.LaMa text generator and OpenAl's CLIP for image embedding, and fine-tuned
16

---

Published as a conference paper at ICLR 2025
Downsies facior:
Room
Groun Truth
DinU-Net
Figure 9: Illustration of tibiae 3D CT auto-segmentation on the Recon-CT dataset from low (left) high (right) resolutions by adjusting the downsizing scale factor. The right tibia is segmented by nnU-Net. Compared with the ground truth, the TotalSegmentator performs better on high resolution CTs.
Pretramed Model
Metical didead (iae-usik)
Training
LLMA
Rol-CT Training Dataset
our Rol-CT & MicoMerdlRee Taatzent
Fine-tured
Fine tuned Model
Sample
Computed tomographhy sean shows the osteolytic hone
lesion an the proximel end of tibia.
Please deeply analyze and diagnose this image.
Generalist Biomedical A. platform
Figure 10: Illustration the training and inference process, and Generative Al platform for diagnosis of breast cancer bone metastasis given 2D cross-scction image from Rol-CT dataset.
with ROCO-dataset (the medical text-image pairs) and our Rol-CT imagery with MiceMediRec text dataset pairs, to generate CT analysis to assist Breast cancer bone metastasis diagnosis (19) (27). As shown in Figure 10, the Web interface demonstrates the ability of fine-tuned LLaVA model to generate the diagnosis given 2D cross- cross-sectional image with breast cancer bone metastasis from Rol-CT dataset.
We have justified the selection of benchmark methods based on their relevance and performance in similar tasks before, as well their compatibility with the BoneMet dataset. Additionally, plan to include more comparative analyses with alternative methods in future work to further evaluate the dataset and benchmarks.
C THE DETAILS OF THREE APIS AND THEIR USAGE EXAMPLES
C.I CT IMAGE SEGMENTATION,
This API provides simple interface to segment the 3D Reconstructed CT (Recon-CT) images into separate CT scans for the spine, left tibia, left femur, right tibia, and right femur. It can handle individual or batched segmentation of the Recon-CT scans. The API reads the 3D CT scans, identifies the appropriate indices to split the images, and saves the segmented scans to the specified output paths. Given the time point (e.g., the week after cancer inoculation), the input folder path, and the
17

---

Published as a conference paper at ICLR 2025
output folder path, Figure 11 exhibits how utilize the CT Image Segmentation API to automatically segment the tibiac.
config
"week": week 0", "masterfolder": r"F: /Recon-CT week
"mastcrout"; r"F: NScg-CTwwok 0"
splitter = ReconcTSsegmentttiicncc (config)
Split a single image
input_folder
folder = F /Recon 01871"
iiage_ttill title = "871"
splitter.split image
image image_title, title, config "masterout"")
Split multiple images
for folder in os. listdir "ocnfigg"masttrrfolee"]]]
if Folder[0:10] in [871, 872, 873, ...]:
input_folder jsipath.join (far
("masterfolder"" folder)
image_title os os.path. bratth.asennme (folder)) [0:12]
splitter.spee iimatt_maageaaa "])
folder, Amage_title, config ("masterout
Figure 11: Example of our CT Image Segmentation API.
config
"workspace"; r"F: \Beg-CT 0",
"cutputdir"; r"F: /Regist-CTV week 0", "refdir": r"F: \reference", "img_z_range", [None, None], "ref_z_range": [None, Nene], "initial saaitiaa_tranffom.. transsorm_angglss:: :ngangle" [np.pi / 16 for in range(-16 10) "BASELINE_REG" True, week (True) or sequencial scans (False)
initialize the registration instance registtationn stteggsstraio (config)
Register a single CT scan input folder = C'F:\Seg-CT\wek. 0"
et id "871 week left tibia"
week
coupptt_fldde contiggooupptddir""
rrggistraiionn... register_ct (input_folder, et_id, week, output_foide:)
Register batch of scans input__flder wer ot_ 1"811 week 0 left tibia", "871 week right tibia", "872 week 11 left tibia", ....
weeek
week = output_folder config("outea ["outputdir"] registration. batchhreegiiiaaae ster (input_folder, ct_ida, week, output_folder)
Figure 12: Example of our CT Image Registration API.
C.2 CT IMAGE REGISTRATION.
This API helps researchers with the tibia registration on Seg-CT dataset. can handle individual or batched registration of the segmented tibiae CTs. The API loads the reference and target CT scans, performs initial transformation, and registers the target CT sean to the reference CT scan. Then the registered CT sean and the transformation saved to the specific output folder. Given the time point
18

---

Published as a conference paper at ICLR 2025
Configuration
config
"foldername"; "selected 300 slices below proximal Tibia-fibula:
junction", Setrst_slice_selette "last slice selected",
"first slice selected",
cesst_sllce_selectee
"first_slies str___sssie__eletteeeeeeeeeeeeee selecteeeedeeeee seelected_belwwtteeeee wtw_bee_tt__funttiite selected slice below the tibia-fibular junction
Index of the first
Initialize the Cropper cropper RolCompos terropperrrrrrreeeeeeeeeee (config)
Crop the RoI from CT images input_folder r"F: /eeiist-CTTibbii w0x5composite" output_foldees os path. join (input_ (input_folderr, (fooderrmmm))
=
last_slicc_selccteed = config( "ttsttsssssts_eeleeeeeeoeoo junetticnn f_junctionn]
cropper.
peoper.ccop_te. crop_roi finput_foldery output_folder first_sl cecst_slice_sst cee_el__eeeeeeeeeeeeeeeeeeeeeeee selecte, _ast_s sirt_ssliee_o Llice_selected first_slice sitst_ssice_belo_ttee below billow_tt_junntile
Figure 13: Example Rol-based CT Image Cropping API.
(e-g., the week after cancer inoculation), the slices range of reference and target subjects, the input folder path, the reference folder path, and the output folder path, Figure 12 illustrates how to utilize the CT Image Registration API automatically align the segmented tibiae.
C.3 ROI-BASED CT IMAGE CROPPING.
This API provides simple interface to crop the region of interest (tibia proximal end) on Regist-CT dataset. It can handle batched cropping of the Regist-CT dataset. The API reads the overlapped 3D Regist-CT composite processed by our python package, identifies the proximal tibia-fibular junction, selects appropriate indices to split the images, and saves the cropped to the specified output paths. Given the input folder path, the output folder path, and index of the first selected slice below the tibia-femoral junction, Figure 13 demonstrates how to utilize the Rol-based CT Image Cropping API to automatically crop the proximal end tibiae.
D BAD-CASE ANALYSIS
Regarding to the bad case analysis, we conducted the following analyses: label noise and data imbalance. Label noise is not concern our primary dataset, as positive and negative labels are assigned the animal level. However, manually created separate dataset with noisy labels. To address label noise in this separate dataset, we applied Label Smoothing (34) with smoothing factor of 0.1, following the DeiT approach (37). The model's accuracy improved from 71.43% (without Label Smoothing) to 95.86% (with Label Smoothing), demonstrating the technique's 's effectiveness in reducing errors associated with noisy labels.
We also examined the effects of data imbalance in our dataset, which had a negative-to-posite ratio of 5:1. To mitigate the challenges posed by this imbalance, implemented Focal Loss (22). The model trained with Cross-Entropy Loss achieved an accuracy of 66.67%, whereas the model trained with Focal Loss improved the accuracy to 69.36%.
E FUTURE POTENTIAL APPLICATION OF BONEMET DATASET
The BoneMet dataset also holds other significant promise for advancing various applications. For example, it can be leveraged to develop foundation models and self-supervised contrastive learning
19

---

Published as a conference paper at ICLR 2025
techniques (49), which will enhance the model's ability to learn robust and generalizable features from the dataset without extensive labeled data. Additionally, the dataset can facilitate the prediction of multi-angle X- X-ray images, providing comprehensive view of bone metastases from different perspectives with reduced radiation exposure and improving diagnostic accuracy and aid in better visualization of complex anatomical structures. Moreover, the BoneMet dataset can be utilized for finite element analysis (FEA) prediction of metastatic bone mechanical properties (31). By integrating FEA with deep learning models, researchers and clinic doctors can easily predict how metastatic lesions affect the mechanical integrity of bones without running the complicated and cone-connu finite element simulation, which crucial for the broad application of finite element analysis in assessing fracture risk and planning appropriate treatments. Lastly, the dataset can be used to train 3D CT registration models, such as the Convolutional Neural Networks (ConvNets) and Deep Learning Image Registration (DLIR) framework (7). These models can accurately align 3D CT scans over time or across different imaging modalities, enabling precise monitoring of discase progression and the effectiveness of treatments.
F ETHICAL STATEMENT
All the animal procedures including cancer implantation and micro CT scans have been approved by the authors' Institutional Animal Care and Use Committee (IACUC), We made every effort to minimize animal suffering throughout the research process. In brief, the pain and stress associated with cancer implantation and cancer growth were carefully monitored (via signs of inflammation, loss of body weight, changes of normal behaviors, and cancer burden) and managed (via administration of painkillers and humane sacrifice). Mice were anesthetized using 3% isoflurane gas, ensuring deep, stable sleep state that prevented movement during surgery and the imaging process with minimized stress or discomfort. Additionally, to reduce radiation exposure, each mouse received 5 weekly micro CT scans of lower limbs using a Bruker/Skyscan in vivo scanner (Bruker 1276). The advantage of this scanner other commercially available in vivo scanners is the low radiation exposure during each session (< 500 mGy) due to the faster scan speed (900 ms) and fewer projections (260). The radiation dose (< 0.5 Gy) is much lower than the lethal radiation dose for mice (10.5 Gy). Our BoneMet dataset adheres to all relevant regulations, including those concerning animal welfare and data protection. Furthermore, we are committed to following guidelines for ethical AI development, including ensuring transparency in Al model development, avoiding bias, and maintaining data privacy, ensuring that the use of this dataset aligns with best practices in responsible Al research.
G LIMITATIONS
Although animal models of breast cancers provide valuable insights into disease development and allow testing the accuracy and sensitivity of various diagnosis methods, we acknowledge the limitations of using animal models in general and the specific model adopted in our study, One potential limitation is the difference in physiology and immune response between mice and humans, leading to the dramatic divergence of the disease progression speed (weeks in mice vs. years in humans). The other limitation is that the mice used in the study were inbred mice with highly homogeneous gene backgrounds, which were maintained under well-controlled living environments. Thus, the animal subjects do not account for the large variability of human diversity regarding gene background, lifestyles (such as diet and physical activities), and underlying health conditions (like Parkinson's ''s disease and obesity). Despite these limitations, the large quantities with detailed labels of animal datasets like ours provide sequential images covering the entire disease development and progression time course, which can be useful test the feasibility and performance of new diagnosis and prognosis tools prior to human trials.
Despite of being limitation, the use of preclinical models is well justified and necessary human cancer research. As elaborated in the "Guidelines for the welfare and use of animals in cancer research" (42), animal experiments remain essential to understand the fundamental mechanisms driving malignancy and to develop individualized molecularly targeted cancer therapies for humans. With the explosion of newly created genetic animal models, the various genes, signaling pathways, and risk factors that give rise to cancer dependencies and vulnerabilities have been identified, and the responses to cancer treatments can be comprehensively tested. Thus animal models and testing not only extend our genetic, molecular, and holistic understanding of cancer but also gather necessary
20

---

Published as a conference paper at ICLR 2025
safety and efficacy information that is required to introduce drugs and diagnostic tools into clinics. Specifically, our dataset was collected from mice implanted with triple-negative breast cancer, which is the most aggressive among all subtypes of breast cancer with limited treatment options. This dataset takes advantage of our animal models, which mimics the full course the disease development from microscopic metastasis lesions macroscopic metastasis (whole bone failure). Currently, we have collected 2D and 3D images of the whole bone and cancer-affected regions at five-time points, along with the biometrics of the animal subjects such their age and body weight. The Dataset is designed to expand and include additional biochemical data such as cancer burden, sera biomarkers, gene transcripts, and histological results. Preclinical research in the cancer field as well as our own studies support the similarities between the mice and humans in terms of the fundamental mechanisms driving cancer spread and growth in the bone environment. Our specific goal is to reveal specific imaging biomarkers associated with cancer growth and bone metastasis, and further make it possible to translate into clinical applications, enabling better understanding, detection, and treatment of bone metastases in human breast cancer.
21


Reviews:
[{"review_id": "zWywr9Un4N", "reviewer": "Reviewer_8mtF", "summary": "This manuscript presents a pioneering large-scale, high-resolution multimodal dataset focused on breast tumor bone metastasis. The dataset encompasses over 50 TB of comprehensive resources across multiple modalities, including 2D X-ray imaging, 3D CT scans, and detailed biological data. The authors have implemented robust accessibility measures through PyPI, Huggingface, and GitHub APIs, ensuring broad availability to the research community.", "questions": "1.\tWhat scientific rationale guided the decision to utilize Swin-base for comparison with ViT, rather than employing ViT with STA for a more direct comparative analysis?\n2.\tHow do the findings from murine breast cancer models specifically inform and advance our understanding of human breast cancer pathophysiology and treatment approaches?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.\tThe dataset represents the first over 50 terabytes-scale multimodal data resources\n2.\tThe implementation of multiple standardized APIs ensures seamless accessibility and integration capabilities\n3.\tThe methodology section provides comprehensive documentation of data collection protocols and analytical procedures\n4.\tThe extensive experimental validation demonstrates the dataset's utility for advanced deep learning applications", "weaknesses": "1.\tThe experimental design exhibits inconsistencies in model selection and parameter settings, potentially impacting comparative analyses, see question 1\n2.\tSeveral, but not limited to, technical inaccuracies require attention: \"detials\" should be \"details\" on line 454, and \"BigThansfer\" should be \"BigTransfer\" in Figure 5\n3.\tThe translational aspects between murine models and human applications require stronger substantiation, particularly given the introduction's emphasis on human breast cancer implications", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8XwkjUZ9uN", "reviewer": "Reviewer_xphi", "summary": "This paper presents a novel large-scale dataset (named BoneMet) of Breast tumor bone metastasis for disease diagnosis, prognosis, and treatment management. It consists of six components: Rotational X-Ray Imagery, Reconstructed CT Imagery, Segmented CT Imagery, Registered CT Imagery, Region of Interest CT Imagery, and Mice Medical Records. Besides, the author conducted a series of experiments on the BoneMet dataset by developing various deep learning models to exhibit its applicability and efficiency in managing BTBM disease. The enormous open-source dataset has significant implications for the development of new algorithms in this field beyond doubt.", "questions": "1. Authors describe that Seg-CT and Recon-CT are independent. What is the difference between 3D CT scans in Recon-CT and Seg-CT? What is the meaning of the adjective 'segmented'?\n2. Line 341 \"Second, we observe a significant training-test accuracy gap of 20.4% with ViT (w/o STA), indicating a pronounced overfitting issue inherent to the ViT architecture.\"\nI think ViT, as a visual backbone network, is less overfitting compared to other structures such as CNN. In addition, is the decrease in accuracy necessarily overfitting?\n3. Section 3.1, the purpose and analysis of experiments is confusing. Why comparing two ViT models could demonstrate the applicability of the Rotation-X-Ray dataset to manage BTBM disease? What is the relation between the effectiveness of\nSTA and the applicability of the Rotation-X-Ray dataset? All subsections 3.2-4 have similar questions.\n4. The experiment did not report data partitioning.\n5. Table 2 reported metrics on the training and testing data. Is there any evaluation/dev data?\n6. The experiment reported many indicator values, but their actual clinical significance has not been evaluated. For example, what are the Precision/Recall/F1-Score/Accuracy of clinical experts on the BTBM diagnosis? Besides, the 3D-GAN, T-VAE, and ST-VAE methods achieve SSIM values of 0.767, 0.817, and 0.860. So, what role can they play in clinical practice?\n7. The authors used many existing softwares as the processing tools to generate the sub-datasets of BoneMet. Please provide the corresponding reference and specify whether it is free of charge or not. Also, please accurately explain in which case the datasets are processed by the Seg-API, Regist-API and RoI-API that were developed by the authors.\n8. Are Pre-Op and Post-Op refer to pre-operation and post-operation? Please give the full spelling.\n9. For RoI-CT, the tibiae ROI is assigned with different pixel values. As shown in Figure 4, for each CT example, is the processed CT image contains the values of 180, 240, 60 and 0 only? If yes, the original CT turns to a segmentation mask instead, and lots of information would be lost.\n10. In the whole preparation procedure, it seems that the manual annotation of the organs or tumors is not involved, except for the manual cropping of fibular by  CTAn®. How could we ensure the effectiveness of the existing software?\n11. In Table 1, the column \"Spatial Resolution\" might be accurately filled out as most of them indicate the organs. Please revise it.\n12. Some details about the BoneMet dataset are missing. For instance, the number of mice is not introduced. Besides, in Table 1, the images/records are classified into two categories, i.e., positive and negative. Is the label assigned at the slice level or animal level? Besides, we noticed that the numbers of Seg-CT are larger than those of Recon-CT, which makes it confusing, as the Seg-CT is obtained from Recon-CT.\n13. I also have concerns about the MiceMediRec dataset. It mainly consists of the medical record and quantitative analysis results of 3D CT images, simulation and mechanical testing. The authors list only a part of the features in Table S2 in the supplementary file. However, all feature should be provided. The off-the-shelf software, such as CTAn®, was used to generate the quantitative parameters, which took about half an hour for each case of animal. In Section 3.3, the multi-modal prognostic assessment model first leveraged the GAN to produce the CT scan of the future point and use the quantitative data in the MiceMediRec dataset as the ground truth label to validate the predicted reaction force values. However, the details about how to obtain the predicted reaction force values are not clear.\n14. In my opinion, multi-modal learning might refer to a way to integrate the information of multi-source data to build a model. However, the application in Section 3.3 just used CT data as model input. Please consider revising it.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "(1) The motivation of this study is clear and the background is well presented in the manuscript.\nThe main contribution of this work is to release a large-scale breast tumor bone metastasis dataset BoneMet, for the research community. Upon the subsets of the BoneMet, the researchers are allowed to develop some novel approaches to solve the task of breast tumor bone metastasis diagnosis and prognosis, and thus facilitating the automated analysis for breast tumor bone metastasis.\n(2) Apart from the information of the collected dataset, the authors also developed several deep learning models to validate the applicability and efficacy of BoneMet.\n(3) Dataset, code and tutorials are made available for free use.", "weaknesses": "(1) The technical contributions of this study are quite marginal, as most of the deep learning models are built upon existing methods.\n(2) Some details about the dataset preparation and experiments are missing.\n(3) Some settings and analyses in the experiment may not support the objectives.\n(4) The quality of the labels, particularly the pixel-wise annotation of the bone, might be questionable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xLq65nxuOl", "reviewer": "Reviewer_1KbZ", "summary": "This paper introduces BoneMet, the first large-scale, open dataset specifically designed for BTBM research. This dataset offers over 50 terabytes of high-resolution, multi-modal data, including 2D X-rays, 3D CT scans, and comprehensive biological records collected from thousands of mice. Structured into six components, the dataset may be suitable for a range of AI tasks focused on BTBM diagnosis, prognosis, and treatment. The authors also conducted extensive experiments to demonstrate the usability of this dataset. The dataset, APIs for flexible data processing and retrieval, accompanying code, and tutorials are freely available. To some extent, this dataset makes a valuable contribution to this field.", "questions": "refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.This paper presents a large-scale, open dataset for the diagnosis and prognosis of breast tumor bone metastasis (BTBM), with a total volume of over 50 terabytes, encompassing various modalities. To some extent, this dataset makes a valuable contribution to this field.\n2.The authors have developed a specialized toolkit for accessing and processing the dataset, facilitating its use for researchers in the field.\n3.The authors conducted detailed experiments that demonstrate the usability of this dataset for some tasks related to BTBM diagnosis and prognosis.", "weaknesses": "1.This dataset is derived from mice, and there are certain differences between mouse skeletons and human skeletons. Can models trained on this dataset be effectively transferred to the diagnosis and prognosis of human BTBM? If applicable, how is the performance? If not, what clinical value does this dataset or model hold?\n2.The authors do not provide sufficient details regarding the experimental setup, particularly how the data was partitioned. The experiments appear to be internal validations conducted solely within the dataset, with no external validation on other datasets. This raises concerns about the generalization ability of the models trained on this dataset. Therefore, it remains uncertain whether this dataset can be used to build versatile large-scale AI models or foundational models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0W0Ef3iDHN", "reviewer": "Reviewer_Ufz4", "summary": "In this study, the authors integrated a publicly available large-scale Bone Metastasis (BoneMet) dataset with multimodal medical data, including 2D X-ray images, 3D CT scans, and comprehensive biological data. The primary objective of this database is to facilitate the diagnosis, prognosis, and treatment management of bone metastasis associated with breast tumors.", "questions": "This paper comprises two main components: the dataset and the benchmark. Both sections could benefit from additional details and comparisons.\n\nThe authors should evaluate and clarify how the collected data can directly benefit human health. For example, they could discuss the potential for training models on their dataset and subsequently transfer the obtained model to human datasets.\n\nFurthermore, the figures in the manuscript require better organization, as the text is currently too small for effective readability.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well-organized and clearly written.\n\nOur collected dataset is open, large-scale, and consists of multi-modal resources including 2D X-ray images, 3D CT scans, and medical records and quantitative analysis.\n\nThe author provides a tool on the Python Package Index (PyPI) for assisting researchers and practitioners.\n\nThe authors have conducted multiple benchmarks, such as BTBM Diagnosis to exemplify the utilisation of the collated dataset.", "weaknesses": "In the abstract, the author states, “Breast tumor bone metastasis (BTBM) affects women’s health globally, necessitating the development of effective solutions for its diagnosis and prognosis.” However, the dataset collected was derived from mice rather than human patients.\n\nThe author should perform a comprehensive comparison between the BoneMet dataset and previously available datasets, including those derived from human subjects, to elucidate the differences and scales involved.\n\nAdditionally, the methods employed in each benchmark were limited. The author should justify the selection of the included methods over others.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)