Package: latte-metrics
Cross-framework Python Package for Evaluation of Latent-based Generative ModelsLatteLatte (forLATent Tensor Evaluation) is a cross-framework Python package for evaluation of latent-based generative models. Latte supports calculation of disentanglement and controllability metrics in both PyTorch (via TorchMetrics) and TensorFlow.InstallationFor developers working on local clone,cdto the repo and replacelattewith.. For example,pip install .[tests]pip install latte-metrics           # core (numpy only)pip install latte-metrics[pytorch]  # with torchmetrics wrapperpip install latte-metrics[keras]    # with tensorflow wrapperpip install latte-metrics[tests]    # for testingRunning tests locallypip install .[tests]
pytest tests/ --cov=latteQuick ExamplesFunctional APIimportlattefromlatte.functional.disentanglement.mutual_infoimportmigimportnumpyasnplatte.seed(42)z=np.random.randn(16,8)a=np.random.randn(16,2)mutual_info_gap=mig(z,a,discrete=False,reg_dim=[4,3])Modular APIimportlattefromlatte.metrics.core.disentanglementimportMutualInformationGapimportnumpyasnplatte.seed(42)mig=MutualInformationGap()# ...# initialize data and model# ...fordata,attributesinrange(batches):recon,z=model(data)mig.update_state(z,attributes)mig_val=mig.compute()TorchMetrics APIimportlattefromlatte.metrics.torch.disentanglementimportMutualInformationGapimporttorchlatte.seed(42)mig=MutualInformationGap()# ...# initialize data and model# ...fordata,attributesinrange(batches):recon,z=model(data)mig.update(z,attributes)mig_val=mig.compute()Keras Metric APIimportlattefromlatte.metrics.keras.disentanglementimportMutualInformationGapfromtensorflowimportkerasastfklatte.seed(42)mig=MutualInformationGap()# ...# initialize data and model# ...fordata,attributesinrange(batches):recon,z=model(data)mig.update_state(z,attributes)mig_val=mig.result()Example NotebooksSee Latte in action with Morpho-MNIST example notebooks on Google Colab:PyTorch (vanilla)PyTorch LightningTensorFlow/KerasDocumentationhttps://latte.readthedocs.io/en/latestSupported metrics🧪 Beta support | ✔️ Stable | 🔨 In Progress | 🕣 In Queue | 👀 KIV |MetricLatte FunctionalLatte ModularTorchMetricsKeras MetricDisentanglement Metrics📝Mutual Information Gap (MIG)🧪🧪🧪🧪📝Dependency-blind Mutual Information Gap (DMIG)🧪🧪🧪🧪📝Dependency-aware Mutual Information Gap (XMIG)🧪🧪🧪🧪📝Dependency-aware Latent Information Gap (DLIG)🧪🧪🧪🧪📝Separate Attribute Predictability (SAP)🧪🧪🧪🧪📝Modularity🧪🧪🧪🧪📝β-VAE Score👀👀👀👀📝FactorVAE Score👀👀👀👀📝DCI Score👀👀👀👀📝Interventional Robustness Score (IRS)👀👀👀👀📝Consistency👀👀👀👀📝Restrictiveness👀👀👀👀Interpolatability Metrics📝Smoothness🧪🧪🧪🧪📝Monotonicity🧪🧪🧪🧪📝Latent Density Ratio🕣🕣🕣🕣📝Linearity👀👀👀👀Bundled metric modules🧪 Experimental (subject to changes) | ✔️ Stable | 🔨 In Progress | 🕣 In QueueMetric BundleLatte FunctionalLatte ModularTorchMetricsKeras MetricIncludedDependency-aware Disentanglement🧪🧪🧪🧪MIG, DMIG, XMIG, DLIGLIAD-based Interpolatability🧪🧪🧪🧪Smoothness, MonotonicityCiteFor individual metrics, please cite the paper according to the link in the 📝 icon in front of each metric.If you find our package useful, please cite open accesspaperonSoftware Impacts(Elsevier) as@article{watcharasupat2021latte,author={Watcharasupat, Karn N. and Lee, Junyoung and Lerch, Alexander},title={{Latte: Cross-framework Python Package for Evaluation of Latent-based Generative Models}},journal={Software Impacts},volume={11},pages={100222},year={2022},issn={2665-9638},doi={https://doi.org/10.1016/j.simpa.2022.100222},url={https://www.sciencedirect.com/science/article/pii/S2665963822000033},}