Package: blctools
BLC Data Analytics tools packageData Analytics helper functions to work with inside BLC's Cloud system.ChangelogVersion 0.0.43DatosCROM()Improved: elaborar_curvas_de_potencia can now accept other dataframes.Version 0.0.43DatosSMEC()Improved: estimar_ENS_xxxx now runs orders between 10x (metodo='cuantil') and 40x (metodo='promedio') faster. Now they both run equally fast.Version 0.0.42DatosSMEC()Fixed bug: Initialization was not working properly.Version 0.0.41DatosCROM()Improved: all functions to estimate ENS were improved drastically. They're now 4x faster.Version 0.0.40DatosCROM()Fixed bug: buscar_indisponibilidades() was not working properlyFixed bug: buscar_limitaciones() was not working properlyFixed bug: buscar_incidencias() was not working properlyVersion 0.0.39DatosCROM()Fixed bug: consolidar_todo() was not categorizing internal/external incidents correctly for some reasonVersion 0.0.38DatosCROM()Fixed bug: exportar() really slowImproved: explotar_incidencias() takes  only incidencts with different start and end dates as inputImproved: Some functions were being executed twice upon object initializationVersion 0.0.37fechasFixed bug: mes_ant_dia_1() wasn't working properly.Version 0.0.36DatosCROM()Fixed bug: incidencias_resumen_status wasn't working properlyTablasVC()Fixed bug: Functions .consultar_.... where not working with "uc" parameterVersion 0.0.35DatosCROM()Added: incidencias_resumen_status attributeDeleted: Client attributeChanged: resumen_diario_incidencias to incidencias_resumen_diarioTablasVC()Deleted: Client attributePPO()Changed: Default dates on initialization are now blctools.ayer()DTE()Changed: Default dates on initialization are now blctools.ayer()Changed: Improved internal modilarityChanged: "dolarizar" parameter to "cols_usd"Added: "dolarizar" methodVersion 0.0.34BCRA()Fixed bug: rem 24 month projections were not being projected properlyVersion 0.0.33BCRA()Added: cargar_tc_min = False as initialization parameterAdded: tc_min as the retail price for the USD in ARSFixed bug: rem projections were not being projected properlyVersion 0.0.32fechasFixed bug: restar_mes wasn't working for some months of the year.Version 0.0.31ArchivoMDB()New object: It serves as an easy way to explore .mdb filesReporteBase()Changed: how the object processes mdb files. It's now more efficientVersion 0.0.30BCRA()Changed: On initializing, rem tryies to download all filesChanged: nor .rem_tc table is called .tc_remChanged: now obtener_rem_mensual() is called cargar_rem()All possible attributes are nor .tc_dia, .tc_udh, .tc_rem and .remDTE()Changed: Cargar DTE goes on even if tc_udh couldn't be loaded from BCRA() objectDatosSMEC()Changed: consultar_datos_vc() is now cargar_datos_vc()Version 0.0.29DTE()Added: Option to convet prices to USDAdded: Initialization parameters cargar (bool), descargar (bool) and dolarizar (str,list,tuple,set).fechasAdded: lista_mensual. Returns a list of dates provided a starting date and an "n"Added: convertir_año_a_fecha() Returns a dt.DateTime object, starting from an integer (year)Fixed bug: mes_ult_dia wasn't allowing for future datesBCRA()Added: Added interface with Argentina's Central Bank to download exchange rate (tc) and Market Expectations (REM)DatosCROM()Fixed bug: After 0.0.25 update, the object wasn't reading .xlsx files properly, thus crashing when trying to read.Version 0.0.28DatosSMEC()Changed: FechaOp column is now datetime type, when Querying VisionCROM's database.Version 0.0.27DatosSMEC()Fixed bug: HoraOp on cargar_datos_prn was not being calculated correctly.Fixed bug: Actaris meters have now the right column order.Changed: Now E_Neta_Gen and E_Neta_Con are calculated when each file is read, for consistency.Version 0.0.26DatosCROM()Added .DatosCROM() now warns if there is an empty excel fileDatosCdO()Added strings reportChanged the variable-reading process, made it more efficient. By adding .pickle file as backup.Added listado_señales_strings(): get list of "string" signals (actually DC combiner boxes)Added descargar_datos_strings(): download "string" signals' mesaurements (actually DC combiner boxes)DatosSMEC()Added this new object to interact with VisionCROM's DataBase and fetch SMEC measurementsInitialization parameters:fecha_i = Data's start date. DateTime object with year, month, day, hour and minute interval.fecha_f = Data's end date. Same as fecha_iparques = CAMMESAS's Nemotécnicos to indicate from which park the measurements will be retrieveddir_salida = export directorycargar_datos_basicos = sames as TablasVC()mensajes = Verbose while loadingmensajes_SQL = Verbose while connectingAdded consultar_datos_vc(): Query VisionCROM's database using fecha_i, fecha_f and parque attributesAdded leer_archivo_prn(file): Read a single .prn fileAdded cargar_datos_prn(folder): Read every .prn file in a given folderVersion 0.0.25TablasVC()Changed: modificar_incidencia() now doesn't affect .incidencias_todas_auditoria attribute.DatosCROM()Improved: .estimar_ENS_varias_incidencias()'s overall speed and precision. Still not good for production, but better for single WTG incidents.Improved: Memory Usage overallRemoved : Pgen_U variable, was a duplicate of FC.Removed: Pcon_U, Egen_U, Econ_U, Epos_U, very rarely useed, could be calculated by hand.DatosCdO()Added this new object to interact with InAccess' API and download raw data from Caldenes del Oeste solar PV Plant (CdO)Initialization parameters:usr = Username (SCADA's username)pwd = PAssword (SCADA's password)apikey = Private apikey provided from InAccessfecha_i = Data's start date. DateTime object with year, month, day, hour and 10 minute interval.fecha_f = Data's end date. Same as fecha_ireporte_produccion = True / False. Indicate if production report should be downloaded automaticallyreporte_trackers = True / False. Indicate if trackers report should be downloaded automaticallyreporte_mensual = True / False. If True, downloads both reports. False has no effect.exportar = True / 'todos' / 'produccion / 'trackers'.If True or 'todos' exports everything.'produccion' and 'trackers' export the respective report each.Added obtener_señales(): Takes in a dataframe of Measurement Locations (Mlocs) and fetches all possible signals.Added obtener_mediciones(): Takes a dataframe of signal lists and fetches dataAdded listado_señales_trackers(): Just a preset, that shows which signals are used for the monthly production reportAdded listado_señales_produccion(): Just a preset, that shows which signals are used for the monthly tracker reportAdded mediciones_disponibles(): Initializing function that loads every available signal, from BLC's cloudAdded descargar_datos_produccion(): Downloads data from listado_señales_produccion() signals using DatosCdO's start and end date parametersAdded descargar_datos_trackers(): Downloads data from listado_señales_trackers() signals using DatosCdO's start and end date parametersAdded exportar(reportes = str): Exports loaded data from preset reports ['todos', 'produccion', 'trackers'].Version 0.0.24DatosCROM()estimar_ENS() now subtracts actual generated power from estimated power lossFixed bug: estimar_ENS() wasn't checking datatypes properlyVersion 0.0.23PPO() and DTE()Fixed bug: Setting "filtro" on initialization had no effect on running functions with default parameters.cl_CarpetaServicios()Added: blctools now finds "Servicios" folder, even when it's not explicitly written in the OS Path.DatosCROM()Added: estimar_ENS() takes a list of ids and estimates the ENS for each incident, based on actual production values.Added: estimar_ENS_incidencias_abiertas() fills the ENS of the incidents that are still openAdded: estimar_ens_inc_abiertas = False initialization parameterAdded: incidencias_a_editar = None, takes a dictionary of dictionaries. The first dict has the IDs of the incidents that have to be edited as keys and their corresponding values are a second dictionary that holds the new parameters (and new values) . Example: {18088:{ENS=800,Owner='NewOwner'},18089:{Hours=8,Origin='INT'}}Fixed bug: Plant Active Power and Energy was being calculated from WTGs / Circuits and it shouldn't. It has it's own valuesExcludes incidents with Hours or ENS < 0 on .incidencias and .incidencias_redux attributes, but not in incidencias_todascargar_incidencias is now False as default.explotar_incidencias is now False as default.Fechas moduleFixed bug: hora_op, fecha_op, sumar_mes / restar_mes now have prevenir_futuro=False as default parameter and pass it to validar_fecha(s)Fixed bug: obtener_periodo now has prevenir_futuro=True as default parameter and passes it to mes_periodo(s)TablasVC()Added .consultar_agrupamiento_de_un_equipo()Added .modificar_incidencia() to edit a current incidentFixed Bug: Query for incidencias was not returning open incidentsVersion 0.0.22DatosCROM()Added: One can now initialize the "parques" parameter with a single string, to avoid using brackets when working with just one park.Fixed bug: Eerror when no active incidents were found for a given parq/period and trying to .interpretar_incidencias_bajo_iec61400()Fixed bug: Some functions where trying to call TablasVC().consultar_incidencias() which doesn't exist anymore. Now its .consultar(incidencias='offline')Version 0.0.21DatosCROM()Added: .buscar_indisponibilidades(). Searches for periods of time which the WTGs met certain condition filters of out of serviceAdded: .buscar_limitaciones(). Searches for periods of time which the PLANT or WTGs had an SP_P > than the max.Added: .buscar_incidencias() combines both previous functions in a single line.Added: .incidencias_autodetectadas attribute, with export capabilitiesAdded: "Status" column to incidencias_reduxAdded: "Count" and "Histogram" columns when calculating powercurves. _count "sheet" on export is now always there.Added: "explotar_incidencias" initialization parámeter, which can take either 'iec61400' (default) or 'crudas'Fixed bug: Missing Indices on datos_seg when trying to consolidate incidents and 10second data_identificar_indisponibilidades() pgen and ppos parameters were added, to further improve filtering capacity__procesar_d10s_dc() and cargar_segundales() now use DataFrame.reindex() to fill with blank rows where necessary.elaborar_curvas_de_potencia() now sorts index clockwise instead of alphabeticallyTablasVC()Added .consultar_agrupamiento_de_un_equipo()Version 0.0.20DatosCROM()interpretar_incidencias_bajo_iec61400() made more efficient, while keeping compatibility with pandas >= 1.5.0_identificar_indisponibilidades() pgen and ppos parameters were added, to further improve filtering capacityTablasVC()Added "potencia=False" parameter to consultar_agrupamientos(), consultar_equipos_no_agrupamientos()Version 0.0.19DatosCROM()Fixed bug: interpretar_incidencias_bajo_iec61400() was experiencing problems with pandas >= 1.5.0, and had a problem with the index selection statement at the beginning.Added experimentally: _identificar_indisponibilidades(duracion=30)Version 0.0.18DatosCROM().exportar() now flattens MultiIndex columns, freezes panes and formats floats, for easier view on Excel.Added "invalidar_datos_seg_congelados" initialization parameter (default = False)Fixed bug: .exportar() throwing error when curvas_de_potencia or consolidado held more than 1 DataframeFixed bug: .interpretar_incidencias_bajo_iec61400() now trims final result between fecha_i and fecha_fFixed bug: .elaborar_curvas_de_potencia() wasn't working properly with filtro_n >1Version 0.0.17fechas submoduleAdded .sem_dia_1(date,offset=0) function. Returns first day of week of the given date + given offset in days (offset can be negative)Added .sem_act_dia_1(offset=0) function. Returns first day of current week + given offset in daysAdded .año_op(fecha_actual,fecha_cod) function. Returns the current contractual year based on the two dates (First year is 1, not 0)DatosCROM()adjusted how xlsx files processing time is calculatedadjusted how .pickle files are created/read. Compression was added through bz2consolidado attribute is now called consolidado_redux, which is a subset of columns of consolidadoconsolidado attribute now holds all the original data, but resampled to the users' input sample rate, excluding categorical variablesconsultar_equipos_parque() and consultar_equipos_parque_no_agrupamientos() are now called consultar_equipos() and consultar_equipos_no_agrupamientos()__iec61400_preparar_df_vacio_minutal() and it's aux methods are now more efficientAdded variables "Gen", "Con", "Pos" as boolean series for each equipment, into the datos_seg dataframeAdded columns "ENS_Lim","ENS_LimExt","ENS_LimInt","ENS_Ind","ENS_IndExt,"ENS_IndInt" to "consolidado" reportAdded variable "Datos_%" which indicates the ammount of not null data per row, per element, into the datos_seg dataframeAdded variable "Registros" which indicates the ammout of rows (measured in time) on the excel file for that given element. Should alwasy equal 24Fixed bug: .__procesar_d10s_5i() wasn't ensuring boolean dtypes for 'Gen','Con' and 'Pos', which impacted negatively on .elaborar_reporte_consolidado()Fixed bug: .consolidar_todo() wasn't working with PEGARCIGFixed bug: .consolidar_todo() wasn't working adding up HOUR columns properly (the rest was fine)Fixed bug: .consolidar_todo() wasn't finishing if there weren't any incidents for a given park on a given time windowFixed bug: .explotar_incidencias() now works properly if there aren't any incidents on a given time windowFixed bug: .exportar() was exporting empty dataframes, since they were not "None"Fixed bug: Roll-up of availability parameters wasn't being done correctly. Now groupings are the mean or logic operator of their groupped elements.Fixed bug: 10s pre-processing: Wind_dir_r was not being saved correctly for 'PLANT' element on self.datos_segFixed bug: 10 second data is now being completed (in case of missing 10second records) with null values. This could have lead to serious problems whenFixed bug: On many methods. From pandas >= 1.5.0 numpy series where linked and not copied when creating a new df. This created many bugs throughout the whole program.unloading incidents to the 10 second data, since the main mechanism behind it is selection trhough .loc[t_stamp,:]TablasVC()Added attributes .habilitaciones and .tipohabilitaciones which are new tables of "datos básicos". They contain COD related information of each plantAdded .consultar_cod(nemo_parque='',uc='') function, to quickly find out about a plant's COD (Birthday) date (grabs from the table above)Added .consultar_año_op(fecha_actual,feca_cod) function. Returns the contractual year (Contractual age) of a power plantAdded .__crear_dict_tipo_empresa(). Which fills in thre internal attributes: _empresas_bop, _empresas_gen, _empresas_gridFixed bug: .consultar() would fetchdatos_basicos from offline file even when 'online' parameter was set. This was due to:a logic error of variable db_off inside .consultar()the order in which instructions were executedFixed bug: __procesar_incidencias_it2() wasn't executing properly when cargar_datos_basicos = 'offline' was set. Solved by calling the new function .__crear_dict_tipo_empresa() before business mappingFixed bug: Desconecting twice from DB when fetching incidenciasFixed bug: __combinar_incidencias_locales_y_online() wasn't working properly. Duplicate dropping should only be done by 'ID'Refactored __asignar_datos() .method, to make it modular per tableVersion 0.0.16DatosCROM()Added data invalidation when processing 10 second data. Now consecutive values of P, Q, Wind, Wind_dir and PPos != 0 and != max (for Ppos) are replaced with np.nanFixed bug when selecting no .parques ( [] ) and initializing a DatosCROM objectFixed bug when truying to export specific reports through the generating functions (ie elaborar_curvas_de_potencia, interpretar_bajo_iec61400, etc.) and seeting the "exportar" parameter to TrueFixed bug. PI wasn't being correctly calculated. (* instead of / was being used)Added "filtro_n" parameter to .elaborar_curvas_de_potencia(), to only show results with an n_count >= n per bin. Default 0 (deactivated)Added feature. Ppos is being calculated if column doesn't exist, based on Curvas_de_potencia.xlsx on BLC's cloud under blc.get_dc_cfg() directory"clientes" initializing parameter was removed (little use for it)"solo_CROM" initializing parameter was removed (little use for it)TablasVC()Improved the processing of incidents, now "Origin" is "NA" and np.nan resistent. Default = "INT"Fixed bug through which a filter for discarded incidents wasn't working.Refactored processing of incidents, it's more readable nowVersion 0.0.15DatosCROM().exportar() was added. Helper function to export all loaded reports to xlsx format..explotar_incidencias() now takes in optional export parameters..incidencias_interpretar_bajo_iec61400() now takes in optional export parameters..incidencias_identificar_solapamientos() was added."curvas_de_potencia" parameter added while initializing, to force power curves to be computed automatically.elaborar_curvas_de_potencia() is now more resistent to failures. Powercurves can be elaborated without having to process incidents before, although not recommended if user doesn't know what he is doing.changed how __renombrar_lvl0() works, now it must be feed the equipment names.TablasVC()fixed a bug, where .consultar_equipos_parque_no_agrupamientos would return "PLANT" twice for GRIOEO"cargar_incidencias" and "cargar_datos_basicos" parameters added while initializing, with possible values being [Flase, True, 'offline', 'online']now queries per default active incidents from the 1st day of two months prior to today (if True or 'online' values where passed to "cargar_incidencias" parameter). Then concatenates obtained results with offline results (if there are any).Version 0.0.14TablasVC()Fixed bug: Corrected functions that took equipments from blctools.TablasVC().conjuntogeneradores, now reading from .tipoequipoFixed bug: Now Parks without any grouping of it's generators can return the generators without grouping. Special Case: PEGARCIGSQLConnector()now decrypts login data from BLC's secure fileserver.DatosCROM().consultar_equipos_parque() now retrieves the right elements if the plant doesn't havy any other grouping stages.consultar_equipos_por_agrupamiento() now returns an empty list if the plant doesn't havy any other grouping stages.rpt_consolidado(granularidad='1D') was added."forzar_reprocesamiento" parameter added while initializing, to force re-process 10 second data..rpt_curvas_de_potencia() was addedchanged the behaviour of .fecha_i and .fecha_i setters. They now attempt to update "incidencias" and "archivos" depending on the values of fecha_i and fecha_f, but the setters dont try to order the dates automatically, which lead to an unwanted behaviour.Changed completely how DatosCROM() now proceses raw data (10 seconds and Incidencias). It's more efficient, and the heavy lifting occurs while processing the 10-second data. Now index has 1 column (Time) and columns have a 3 level structure (p,e,v) = Park, Element, VariableVersion 0.0.13Created SQLConnector class to migrate to SQLAlchemyblctools.TablasVC now inherits from SQLConnectorVersion 0.0.12Function blctools.fechas.obtener_periodo() was added (moved from blctools.ReporteBase())Added more parameter flexibility within all functions inside blctools.fechas.Objects PPO(), DTE() and DatosCROM() now accept the "periodo" initialization parameter (overrides fecha_i and fecha_f).bjects blctools.DatosCROM() and blctools.TablasVC() have now the "solo_CROM=False" initialization parameter.function blctools.TablasVC.consultar_nemoCammesa() has now the "solo_CROM=False" argumentVersion 0.0.11Corrected blctools.PPO.periodo('mes_anterior')Function ReporteBase.a_excel() accepts now the parameter "exportar_consulta=bool"Objects PPO() and DTE() now accept the following initialization parameters"filtro""periodo" (overrides fecha_i and fecha_f)No more unnecessary warnings on changing the "filtro" parameterFunction blctools.fechas.mes_dia_1() and blctools.fechas.mes_ult_dia() now validate dates using blctools.fechas.validar_fechas()Functions blctools.fechas.validar_fecha() and .validar_fechas() now have a parameter "prevenir_futuro=True" as default, to prevent future dates or not.Fixed a bug, when changing "filtro" on functions of child classes of DTE() or PPO(), download and extraction directories wouldn't update properly.Version 0.0.10Attribute blctools.DatosCROM.dir_salida sets to which directory the reports go out to (didn't have much impact before)Improved automatic Servicios directory searchVersion 0.0.9Removed dependencies listAbility to download unfiltered PPOs/DTEs without specifying parks/plants listIncidents are no longer loaded by default on blctools.DatosCrom() objectsVersion 0.0.8Corrected dependencies listVersion 0.0.6 ~ 0.0.7Bug fixes regarding file managementAbility to customize filters when hitting CAMMESA's APIPBA calculation according to IEC61400 is still functioning incorrectlyVersion 0.0.5Most of the code is Object Oriented now.CAMMESA's forecasts have been added.Version 0.0.2 ~ 0.0.4Fixed the install issuesVersion 0.0.1Basic functionality is up and runningTO DOsCorroborar que la cantidad de grupos de columnas en los archivos 10 segundales, coincida con la cantidad de equipos del parque cuyo nombre de archivo enuncia.Convertir DatosCROM().parques a "nemos" automáticamente.Mejorar los mensajes de procesamiento nivel 0, 1 y 2 en todas las funcionesFix Bug: Code could break by passing an '' element into parques initialization parameter